#2881 util::CsvInStream readAllrows

SpaceGhost Sat 17 Dec 2022

  1. I am working through reading and writing from text and csv files.
  2. The script below successfully reads input.txt into a string and echoes it to the screen. It then writes the what was read in to an output.txt file.
  3. It also reads in a third file called input.csv into a Str[] and echoes the first first three rows to the screen delaminated correctly.
  4. It is the last four lines of the block below where I try to read in the same input.csv file into a 2D Str[][] that I get this error:
Unknown slot 'sys::File.CsvInStream' 
  • I realize I am using the util pod and not the sys pod and using CsvInStream and not InStream (in). I read the docs, examples, and the source in the docs....
  • I have a feeling this is a syntax issue but I cannot figure it out. It is my second day learning Fantom and I need some help. I really enjoy Fantom...what an elegant language with very intelligent syntactic sugar.
  • Respectfully, Space Ghost
class ReadString
{
  public static Void main (Str[] args)
  {
    Uri fileuri := `./src/input.txt`
    echo(fileuri)
    echo(fileuri.typeof)

    Str contents := File(fileuri).readAllStr
    echo ("contents: $contents")

    Uri fileuri2 := `./src/output.txt`
    fileuri2.toFile.out(false).printLine(contents).close

    Uri fileuri3 := `./src/input.csv`
    Str[] contentscsv := File(fileuri3).in.readAllLines
    echo; echo(contentscsv[0..2])

    Uri fileuri4 := `./src/input.csv`
    Str[][] contentscsv2 := File(fileuri4).CsvInStream.readAllRows
    echo;echo
    echo(contentscsv2[4])
  }
}

Henry Sat 17 Dec 2022

Hi Space Ghost!

To Construct the CsvInStream class you need to call the make constructor with an InStream, so it's not possible to create it inline using the dot operator as you've shown here.

It's only a small Syntactical change to get this working, try something like:

Uri fileuri4 := `./src/input.csv`
Str[][] contentscsv2 := CsvInStream(File(fileuri4).in).readAllRows
echo;echo
echo(contentscsv2[4])

For some context on the compilation error you're seeing, it's because of the dot operator, it's looking for a method or field on File called CsvInStream (Which doesn't exist!)

Hope that helps!

SpaceGhost Sat 17 Dec 2022

Hi Henry, thank you so much for the reply. Your explanation makes sense and below is the code change. It does not work , rather this provides another compilation error.

  1. I used the dot operator .readAllRows at the end of line(12) which is a slot method for the CsvInStream class to read in the wrapped stream.
  2. The (File(fileuri2)in) in line(12) is the InStream that the CsvInStream should use make to wrap it and make an instance (object) of a CsvInStream that we can read in with the appropriate method....right? BTW, lines (6), (7), and (8) work perfectly but they are part of the sys::File abstract const class and not the util::CsvInStream class
  3. Here is the compilation error which I commented in above the suspect line (12) of code as well:
readcsv.fan(12,26): Unknown method 'readcsv_0::ReadCsv.CsvInStream'
ERROR: cannot compile script
  • I tried a half dozen variations of this with no luck??? I read the source multiple times and this all looks like it should work.... hope someone can help .
1  class ReadCsv
2  {
3   public static Void main (Str[] args)
4   {
5     // the two lines below work and echo csv contents as Str[]
6     Uri fileuri := `./src/input.csv`
7     Str[] contents := File(fileuri).readAllLines  // convenience for .in.readAllLines, guarenteed close of stream
8     echo(contents)
9         
10    Uri fileuri2 := `./src/input.csv`
11    // the line below produce this error: Unknown method 'readcsv_0::ReadCsv.CsvInStream' ERROR: cannot compile script
12    Str[][] contents2 := CsvInStream(File(fileuri2).in).readAllRows
13    echo(contents2)
14  }
15 }

SlimerDude Sat 17 Dec 2022

Hi SpaceGhost,

By default, the Fantom compiler only knows about classes in the sys pod. But CsvInStream is a class in the util pod - so we have to tell the compiler where to find it. Do this with a using statement:

using util::CsvInStream

class ReadCsv {
    public static Void main (Str[] args) {
        ...
        ...
    }
}

See Using in CompilationUnits for more details.

SpaceGhost Sat 17 Dec 2022

One new question below is (5). The rest is a thank you and showing what worked to benefit future readers of this thread.

  1. First off, thank you to SlimerDude and Henry! Your both really helped me get over the hump...and with a little more sweat-and-reading of the doc library source and reviewing the * Examples * it is all sorted out. For my next step, I am going to do binary read/write next, LOL....so may run into some questions again!
  2. For the record - I spent most my life as (in order) a veteran of: BASIC, PASCAL, FORTRAN, C, D, and a touch of NIM -- all Procedural only, i.e. Java / Fantom is my first Object Oriented experience...interesting and I am going between WTH! and This is Cool! I started with Java three weeks ago and said wow, this OO is complex and then found Fantom a few days ago from SlimerDude's couple videos that read my mind on Java which has the gold standard SDK and runtime (JVM) but is a beast.
  3. Second, I am going to post a long block below summarizing my learning, but also as a backdrop to one additional question in (5) below.
  4. You will see below that I was able to read/write my Str[][] list using the util:CsvIn(Out)Stream class and methods. To write the Str[][] back to disk I used a for loop with iterator and cycled through the size of the Str[][].
  5. My question is there a way or pod::Class.method that I am missing to do this in one "gulp"?, i.e. without using a for loop at the end of the .fan file.
  6. For below: I will first paste the:
    • (a) input.txt file which is of course the same as the input.csv file you see the src opening up steams to,
    • (b) then paste the .fan file,
    • (c) then I will paste in the output.txt, and
    • (d) outputcsv.txt files so you can see that the former (with sys::Uri.toFile.out.printLine) done by line has blanks (between comma deliminator) for the cells that were missing in the original input data files (I did this on purpose to see how the methods handled them and they did great), and the latter (with util::CsvOutStream.out.writeRow) inserts the blanks as "" in the disk file.

Respectfully, SpaceGhost

(a) INPUT.TXT (INPUT.CSV is the same, I just changed the file suffix) FILES

Number,Color,Food,Day,Date
1,green,apple,Mon,1
2,red,beans,Tue,3
3,,milk,,9
4,brown,coffee,Thu,15
5,pink,onion,Fri,24
6,yellow,chips,Sat,31
7,purple,,Sun,
8,orange,butter,,14
9,silver,bread,,22
10,gold,tea,,28

(b) .fan FILE

using util::CsvInStream
using util::CsvOutStream

class ReadString
{
  public static Void main (Str[] args)
  {
    echo
    Uri fileuri1 := `./src/input.txt`

    Str contents := File(fileuri1).in.readAllStr // self-closes
    echo ("Str contents from -- sys::File readAllStr: \n$contents"); echo

    Uri fileuri2 := `./src/output.txt`
    fileuri2.toFile.out(false).printLine(contents).close
    // output.txt is from sys::Uri.toFile.out.printLine

    Uri fileuri3 := `./src/input.csv`
    Str[] contentscsv := File(fileuri3).in.readAllLines // self-closes
    echo("Str[] contentscsv from -- sys::File readAllLines: \n$contentscsv"); echo

    Uri fileuri4 := `./src/input.csv`
    Str[][] contentscsv2 := CsvInStream(File(fileuri4).in).readAllRows // self-closes
    rows := contentscsv2.size; cols := contentscsv2[0].size
    echo("contentscsv2 number of rows: $rows")
    echo("contentscsv2 number of columns: $cols\n")
    echo("Str[][] contentsscv2[][] from -- util::CsvInStream readAllRows: \n${contentscsv2[0..rows-1]}"); echo
   
    Uri fileuri5 := `./src/outputcsv.txt`
    File(fileuri5).delete
    for (i:=0; i<rows; i++)
    {
      CsvOutStream(File(fileuri5).out(true)).writeRow(contentscsv2[i]).close
    }  // outputcsv.txt is from util::CsvOutStream.out.writeRow
  }
}

(c) output.txt

Number,Color,Food,Day,Date
1,green,apple,Mon,1
2,red,beans,Tue,3
3,,milk,,9
4,brown,coffee,Thu,15
5,pink,onion,Fri,24
6,yellow,chips,Sat,31
7,purple,,Sun,
8,orange,butter,,14
9,silver,bread,,22
10,gold,tea,,28

(d) outputcsv.txt

Number,Color,Food,Day,Date
1,green,apple,Mon,1
2,red,beans,Tue,3
3,"",milk,"",9
4,brown,coffee,Thu,15
5,pink,onion,Fri,24
6,yellow,chips,Sat,31
7,purple,"",Sun,""
8,orange,butter,"",14
9,silver,bread,"",22
10,gold,tea,"",28

End of Reply

Henry Sat 17 Dec 2022

Glad to see it's working!

Regarding your question, The CsvOutStream class unfortunately doesn't contain a method like writeRows or something that would be able to write the csv wholesale, but I would instead recommend that you first construct an instance of CsvOutStream with the File.out and then simply loop over your list of contentscsv2 using the List.each method, calling writeRow with each item in the list.

Something along the lines of:

Str[][] csvIn := CsvInStream(File(`./src/input.csv`).in).readAllRows
outFile := File(`./src/outputcsv.txt`)
outFile.delete
outStream := CsvOutStream(outFile.out(true))
csvIn.each |Str[] row| {
  outStream.writeRow(row)
}
outStream.close

This way you save the requirement of calculating the size of the list to loop over (as each simply iterates over each item in the list) and also means you will only have to create and open/close the one instance of the CsvOutStream for writing!

One other tiny think is that generally, unless you need the Uri itself for something else, when working with File IO, I'd recommend just constructing the File instances inline with the uri (but that's only really a preference for code cleanliness!)

There will be some OutStream subclasses that will allow a full write of rows, though they tend to primarily appear in the haystack classes for SkySpark, where they make heavy use of Grids, and more List-oriented objects. But in essence what they will do behind the scenes is just the same as calling an each on a list of items and writing them one by one.

SpaceGhost Sat 17 Dec 2022

Henry, you are so right. With the open and close for a large CSV the script took 73 seconds. By moving it outside the for-each or for-next loop we dropped below 3 seconds!. FYI, the .size read for row and column on took literally 1ms more time.

I just posted #2882 (https://fantom.org/forum/topic/2882)with some very cool results using your suggestion and then I did the same with a for-next in Fantom. I compared these results with a huge csv to Basic (and indirectly with C,D,and Fortran which were slightly slower than Basic). Take a look...pretty neat.

Cheers!

Login or Signup to reply.