Shell scripting, help me with what should be a simple task?!

Commissario
Joined
23 Nov 2004
Posts
41,851
Location
Herts
I need to create a lot of files for testing, however I've failed at the first hurdle! Here's what I've set out to achieve and broken down into tasks:
  1. Read a CSV file containing a few thousand rows of data (only one column)
  2. Create a directory with a unique name (in the format X_DD-MM-YYYY_HH-MM-SS) eg. 1_24-06-2018-00-00-01 - hoping I can increment the time value to make the dir unique?
  3. Grab the value from the CSV and store it as a variable
  4. Create a .json file with the above variable used once in one of the json fields and the directory name in one of the json fields
  5. Save the .json file in the above dir
  6. Copy a PDF file from another dir (I should be able to do this using the 'cp' function)
  7. Zip up the new folder with the .json file and the PDF file
I've got this so far but when I run it, nothing happens :(

Code:
while IFS="," read -r f1
do
    #create a unique int that increments through each loop
    int i=1

    #print the case number from each row
    echo $f1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i

done < CCD_CaseList_01-10-2018_1only.csv

I have cd'd into the correct dir before the above code btw.

Can anyone help please?!
 
Man of Honour
Joined
30 Oct 2003
Posts
13,229
Location
Essex
I have a vb.net solution for xml which you could probably adapt, it's doing similar with xml files and json serialization. I built it to add a reporting layer to an app that is buiIt on xml but offered no reporting. I can dig around and share it later if you run into a block on this shell script.

Fwiw a .net solution can easily do all of the above and I can't see anything in there that is particularly difficult to write.
 
Last edited:
Man of Honour
Joined
30 Oct 2003
Posts
13,229
Location
Essex
I have run into a block :(

you got any examples and ill throw it together for you? Shouldn't take that long at all, everything bar zipping up I already have code snippets for. Do you have access to visual studio?

By examples I mean an example of what the csv and finished json might look like?

point 2 you can use the clock to randomise but you could just increment using a flat file to store current file directory number and increment from there, everything else is trivial, im sure even zipping is just a system.io call but have just never needed it.

Also how often do you need to do the task? Edit: In fact it doesn't really matter ill throw something together later this evening/tomorrow morning and post it up you can then change it and away you go.
 
Last edited:
Commissario
OP
Joined
23 Nov 2004
Posts
41,851
Location
Herts
you got any examples and ill throw it together for you? Shouldn't take that long at all, everything bar zipping up I already have code snippets for. Do you have access to visual studio?

By examples I mean an example of what the csv and finished json might look like?

point 2 you can use the clock to randomise but you could just increment using a flat file to store current file directory number and increment from there, everything else is trivial, im sure even zipping is just a system.io call but have just never needed it.

Also how often do you need to do the task? Edit: In fact it doesn't really matter ill throw something together later this evening/tomorrow morning and post it up you can then change it and away you go.
The CSV is many lines of just numbers, it's data that I need to put into the json file, I will PM you an example of it because I can't post it on here.
 
Don
Joined
5 Oct 2005
Posts
11,143
Location
Liverpool
Code:
while IFS=, read -r a1
do
    #create a unique int that increments through each loop
    i=1

    #print the case number from each row
    echo $a1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i
done < no.csv

That works for me mate... let me know if it doesn't tested in Linux (ubuntu)

Stelly
 
Commissario
OP
Joined
23 Nov 2004
Posts
41,851
Location
Herts
Code:
while IFS=, read -r a1
do
    #create a unique int that increments through each loop
    i=1

    #print the case number from each row
    echo $a1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i
done < no.csv

That works for me mate... let me know if it doesn't tested in Linux (ubuntu)

Stelly
Nope, nothing happens in terminal (running on Mac OS Mojave).
 
Don
Joined
5 Oct 2005
Posts
11,143
Location
Liverpool
Code:
#!/bin/sh
while IFS=, read -r a1
do
   #create a unique int that increments through each loop
    i=1

    #print the case number from each row
    echo $a1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i
done < no.csv

Also, can you try giving execute rights to the script:

chmod u+x <whateverscriptname.sh>

Is it giving you no feedback at all? It is the reason I'm thinking its not executing at all...

Stelly
 
Don
Joined
5 Oct 2005
Posts
11,143
Location
Liverpool
When I run it I get the following:

Code:
stelly@DORA:/mnt/d/Maccy$ sh code.sh
1
2
3
4
5
6
7
8
9
10
stelly@DORA:/mnt/d/Maccy$ ls
1_24-06-2018-00-00-1  code.sh  no.csv

Stelly
 
Commissario
OP
Joined
23 Nov 2004
Posts
41,851
Location
Herts
Code:
#!/bin/sh
while IFS=, read -r a1
do
   #create a unique int that increments through each loop
    i=1

    #print the case number from each row
    echo $a1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i
done < no.csv

Also, can you try giving execute rights to the script:

chmod u+x <whateverscriptname.sh>

Is it giving you no feedback at all? It is the reason I'm thinking its not executing at all...

Stelly

Yeh no feedback at all. I changed the script to just a simple mkdir command and that worked so not sure what's going on.

@AHarvey doesn't have to be, no, but it's the only scripting language I'm somewhat familiar with (or apparently not) that I can use on my Mac.
 
Man of Honour
Joined
30 Oct 2003
Posts
13,229
Location
Essex
Not far off of an .exe solution or at least a full .sln file you can play with, question... the csv file... loads of columns or just a single row/column of data?

My apologies I have only just started tbh, this morning I ended up doing an order of service for a family funeral on thurs :(
 
Commissario
OP
Joined
23 Nov 2004
Posts
41,851
Location
Herts
Not far off of an .exe solution or at least a full .sln file you can play with, question... the csv file... loads of columns or just a single row/column of data?

My apologies I have only just started tbh, this morning I ended up doing an order of service for a family funeral on thurs :(
I do have a Windows 10 Parallels instance installed. And the CSV is just a single column with thousands of rows.

Appreciate the help mate, sorry to hear that :(
 
Man of Honour
Joined
30 Oct 2003
Posts
13,229
Location
Essex
I do have a Windows 10 Parallels instance installed. And the CSV is just a single column with thousands of rows.

Appreciate the help mate, sorry to hear that :(

K i'm just building some modules and stuff and if nothing comes up it shouldn't bee too long.
 
Soldato
Joined
6 Mar 2008
Posts
10,078
Location
Stoke area
@Maccy, can you trust me over an example csv and json file?

As it's a mac s python script could do the same thing without the need for an exe.

Am I right in thinking that it's a custom folder, with a single json file made from the csv, a pdf copied over and then zipped?

we'r e not talking 1 csv row = 1 json file = one zip?
 
Don
Joined
5 Oct 2005
Posts
11,143
Location
Liverpool
Okay, here is some powershell you can run in Windows 10 mate:

$p.Number = Number is the name of the column within the csv...

Code:
$p = Import-Csv D:\Maccy\no.csv

$i=0

foreach ($no in $p.Number)
{
    Write-Host $no

    New-Item -ItemType directory -Path "D:\Maccy\1_24-06-2018-00-00-$i"

    Set-Location -Path "D:\Maccy\1_24-06-2018-00-00-$i"

    $i++
}
 
Commissario
OP
Joined
23 Nov 2004
Posts
41,851
Location
Herts
@Maccy, can you trust me over an example csv and json file?

As it's a mac s python script could do the same thing without the need for an exe.

Am I right in thinking that it's a custom folder, with a single json file made from the csv, a pdf copied over and then zipped?

we'r e not talking 1 csv row = 1 json file = one zip?
I'll PM you details.

And yes, it's one zip per line of data in the csv containing the json file (with the 1 bit of data from csv) and pdf.
 
Man of Honour
Joined
30 Oct 2003
Posts
13,229
Location
Essex
well I have quite a lot of it done, reading the csv into an array etc is all done. now it's just a case of formatting the json and pushing the zip etc out. This would end up basically being a form where you set your input and output directory and away you go. Do i still finish it? Right now it just pops up txt boxes at each step with the csv location, full array, chopped up array. The rest is fairly easy stuff.

Code:
'Set up our imports
#Region "Imports directives"
Imports System.IO
Imports System
Imports System.Text
#End Region
Public Class Form1
    Dim CSVFileCtnts As String
    Dim MainRoot As String
    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        ReadCSV()
        ReadCSVFileToArray()
    End Sub
    Private Sub ReadCSV()
        'Variable Declerations
        'Our root directory where everything starts
        Dim root As New DirectoryInfo("C:\tempTest")
        'we can use root along with files to pick any csv in the folder - We shall do this so we don't have to be specific.
        'Files is the files available that are of type csv in the root
        Dim Files As FileInfo() = root.GetFiles("*.csv")
        Dim FileName As FileInfo
        Dim sourceFile As String
        Dim tempFile As String = ("c:\tempTest\temp.txt")
        Dim outputFile As String = ("C:\tempTest\output.txt")
        Dim currentRow As String = ""
        Dim StreamWriter As StreamWriter
        For Each FileName In Files
            Try
                'Try loop - There is only one csv file in the temp directory - This grabs the csv file of any name and reads everything
                'into a single string, we will chop it up later.
                sourceFile = (root.ToString & "\" & FileName.ToString)
                MainRoot = sourceFile
                MsgBox(MainRoot)
                Using streamReader As New Microsoft.VisualBasic.FileIO.TextFieldParser(sourceFile)
                    currentRow = streamReader.ReadToEnd()
                    'Close the streamReader
                    streamReader.Close()
                End Using
                'This is where we split each entry and store the output into an array
         
                'Write the current line to the streamwriter
                StreamWriter = File.AppendText(outputFile)
                StreamWriter.WriteLine(currentRow)
                CSVFileCtnts = currentRow
                'Close the stream writer.
                StreamWriter.Flush()
                StreamWriter.Close()
            Catch ex As Exception
            End Try
            MsgBox(CSVFileCtnts)
        Next
    End Sub
    Private Sub ReadCSVFileToArray()
        Dim strfilename As String
        Dim num_rows As Long
        Dim num_cols As Long
        Dim x As Integer
        Dim y As Integer
        Dim strarray(1, 1) As String
        ' Load the file.
        strfilename = MainRoot
        'Check if file exist
        If File.Exists(strfilename) Then
            Dim tmpstream As StreamReader = File.OpenText(strfilename)
            Dim strlines() As String
            Dim strline() As String
            'Load content of file to strLines array
            strlines = tmpstream.ReadToEnd().Split(Environment.NewLine)
            ' Redimension the array.
            num_rows = UBound(strlines)
            strline = strlines(0).Split(",")
            num_cols = UBound(strline)
            ReDim strarray(num_rows, num_cols)
            'Debug
            ' Copy the data into the array.
            For x = 0 To num_rows
                strline = strlines(x).Split(",")
                For y = 0 To num_cols
                    strarray(x, y) = strline(y)
                Next
            Next
            'Debug
            ' Display the data in messageboxes
            For x = 0 To num_rows
                For y = 0 To num_cols
                    MsgBox(strarray(x, y))
                Next
                MsgBox(Environment.NewLine)
            Next
        End If
    End Sub
End Class



I will carry on but if you prefer a powershell solution over a .net app then let me know. Obviously I would give you all the source code and either pre compile or you can compile yourself... Im easy going... Also I love the way the code box has annihilated what was relatively nicely formatted code :(

edit: it now makes a directory, json file etc... gonna head home for the day but will finish it when I get home. Literally I think all I need to do is zip at the end.
 
Last edited:
Back
Top Bottom