[eiffel_software] Raw or binary structures

[Joe Abbate [e-mail] [eiffel_software]]

Hello,

Is it possible to declare a binary structure in Eiffel, i.e., one that
corresponds to a fixed length record in a RAW_FILE, and in particular
one that may contain fixed length, possibly zero-terminated strings?

From what I've read, one can declare a "raw" structure that contains
single characters, integers and reals, e.g.,

struct: TUPLE [CHARACTER_8, INTEGER_32, REAL]

Then, presumably if one were to use RAW_FILE read_stream (9) one could
read a fixed length record into RAW_FILE's last_string. However, it
would still be necessary to split apart last_string and assign it to the
TUPLE components, or perhaps directly to a regular Eiffel class.

FIRST QUESTION: Can one read from a RAW_FILE directly into such a TUPLE?
Also, can a TUPLE be expanded, and can the expanded size (i.e., the 9
above) be obtained somewhere?

However, how can one declare a C-style string, e.g., something like a

char text[16];

I know that ARRAY maintains its data in a SPECIAL [G] attribute, but
declaring a SPECIAL [CHARACTER_8] element in the tuple above would not
allocate 16 bytes. One could create a TEXT_16 class whose creation
routine would allocate the 16 bytes, but I think that still wouldn't
help because what's really needed is an expanded class.

SECOND QUESTION: Is there a way to declare something equivalent to a
char [n] where n is a fixed value?

FINAL or COROLLARY QUESTION: Is there a way to do what I describe above
purely in Eiffel, or is it necessary to descend to a C interface?

Thanks for your feedback,

Joe

[ljr1981 [e-mail] [eiffel_software]]

I think some of what you're after can be found in the gobo kernel library:
KI_BINARY_INPUT_FILE

For myself, we had to create a library that knows how to read and write
standard xBase DBF files (specifically FoxPro), because we share data with
legacy FoxPro systems. There, data must be converted before writing the
bytes to the file and when reading them back. The file itself contains its
own specification about the structure of the data in a header area of the
DBF structure. For example, the specification for writing integers is
reverse order to how integers are represented in memory. So, they must be
broken down, reversed, and then written to the file. Reading back is the
opposite algorithm.

As for writing complex data structures, you might try redefinition of the
`out' feature (from {ANY}) such that a write to a {RAW_FILE} is as simple as
calling `out' on your complex object graph and writing the {STRING} to the
{RAW_FILE}. If the notion of "records" will always be conforming, then
reading back in might be as simple as known EOR marker, other than newline.

You also have the option for writing to a transportable and self-describing
structure like JSON. We use JSON extensively for writing free-form object
graph data structures to and from disk files. It makes me wonder if we ought
to make our JSON extension library open source. After almost 5 years, we've
beat most of the kinks out and have a fairly stable library.

There are even better binary forms of JSON and even less noisy file
structures like TOML and Pattern Buffers. Yet, I have to agree with Dr.
Dobbs that YAML is just about the best I have seen for brevity. It's a
little complex, but the overhead structure specification is minimal, which
maximizes the data capacity. If one adds binary compression, then maybe YAML
(or something like it) is the end of the line.




-----
Larry Rix
--
View this message in context: http://eiffel.641255.n2.nabble.com/Raw-or-binary-structures-tp7582721p7582722.html
Sent from the Eiffel Software Users mailing list archive at Nabble.com.

[Joe Abbate [e-mail] [eiffel_software]]

Post by ljr1981 ***@msn.com [eiffel_software]
There are even better binary forms of JSON and even less noisy file
structures like TOML and Pattern Buffers. Yet, I have to agree with Dr.
Dobbs that YAML is just about the best I have seen for brevity. It's a
little complex, but the overhead structure specification is minimal, which
maximizes the data capacity. If one adds binary compression, then maybe YAML
(or something like it) is the end of the line.

Thanks, Larry. I'll have to check your recommendations.

Although this is unrelated to Eiffel, I have to agree on YAML. It's
much better that the XML used for the .ecf files. My open source
project (https://github.com/perseas/Pyrseas ) uses YAML to store almost
the complete set of Postgres catalogs.

Cheers,

Joe


------------------------------------

------------------------------------

------------------------------------------------------------------------
Eiffel Software
http://www.eiffel.com
Customer support: http://support.eiffel.com
User group: http://groups.eiffel.com/join
------------------------------------------------------------------------
------------------------------------

Yahoo Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/eiffel_software/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/eiffel_software/join
(Yahoo! ID required)

<*> To change settings via email:
eiffel_software-***@yahoogroups.com
eiffel_software-***@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
eiffel_software-***@yahoogroups.com

<*> Your use of Yahoo Groups is subject to:
https://info.yahoo.com/legal/us/yahoo/utos/terms/

[Jocelyn Fiat [e-mail] [eiffel_software]]

Hi,

Did you had a look at storable solution, and recommended SED solution?
(Old post on https://room.eiffel.com/node/393 )

Using SED, and features from SED_STORABLE_FACILITIES

Here is code to demonstrate quickly how to use SED to store and load Eiffel
object to and from a medium (here, a file).

class TEST
inherit
SED_STORABLE_FACILITIES
create
make

feature -- Access
make
local
f: RAW_FILE
l_writer: SED_MEDIUM_READER_WRITER
l_reader: SED_MEDIUM_READER_WRITER
l_data: TUPLE [i: INTEGER; s: STRING]
do
l_data := [123, "test"]

-- Store `l_data'
create f.make_with_name ("data.bin")
f.open_write
create l_writer.make_for_writing (f)
store (l_data, l_writer)
f.close

-- Retrieve Eiffel object from file "data.bin".
create f.make_with_name ("data.bin")
f.open_read
create l_reader.make_for_reading (f)
if attached {TUPLE [i: INTEGER; s: STRING]} retrieved
(l_reader, True) as d then
print ("i=" + d.i.out + "%T s=" + d.s + "%N")
end
f.close
end

end

Hope this helps,
-- Jocelyn

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]

Post by ljr1981 ***@msn.com [eiffel_software]
There are even better binary forms of JSON and even less noisy file
structures like TOML and Pattern Buffers. Yet, I have to agree with Dr.
Dobbs that YAML is just about the best I have seen for brevity. It's a
little complex, but the overhead structure specification is minimal,

which

Post by ljr1981 ***@msn.com [eiffel_software]
maximizes the data capacity. If one adds binary compression, then maybe

YAML

Post by ljr1981 ***@msn.com [eiffel_software]
(or something like it) is the end of the line.

Thanks, Larry. I'll have to check your recommendations.
Although this is unrelated to Eiffel, I have to agree on YAML. It's
much better that the XML used for the .ecf files. My open source
project (https://github.com/perseas/Pyrseas ) uses YAML to store almost
the complete set of Postgres catalogs.
Cheers,
Joe
------------------------------------
------------------------------------
------------------------------------------------------------------------
Eiffel Software
http://www.eiffel.com
Customer support: http://support.eiffel.com
User group: http://groups.eiffel.com/join
------------------------------------------------------------------------
------------------------------------
Yahoo Groups Links

--
Jocelyn
------------------------------------------------------------------------
Eiffel Software
805-685-1006
http://www.eiffel.com
Customer support: http://support.eiffel.com
User group: http://groups.eiffel.com/join
------------------------------------------------------------------------


[Non-text portions of this message have been removed]

['Emmanuel Stapf' [e-mail] [eiffel_software]]

Hi

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
Is it possible to declare a binary structure in Eiffel, i.e., one that
corresponds to a fixed length record in a RAW_FILE, and in particular one that
may contain fixed length, possibly zero-terminated strings?

Yes it is possible and the way you do it is by manually controlling how bytes are read/write, not via the type system.

Using your example of declaring

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
struct: TUPLE [CHARACTER_8, INTEGER_32, REAL]

one could build an infrastructure to read exactly 9 bytes. This would involve some introspection, but I can see someone doing this very easily with little overhead (meaning no splitting).

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
FIRST QUESTION: Can one read from a RAW_FILE directly into such a TUPLE?
Also, can a TUPLE be expanded, and can the expanded size (i.e., the 9
above) be obtained somewhere?

You can only read basic information using RAW_FILE. To read objects, you can use the storable mechanism of IO_MEDIUM or the one provided by the SED classes. But this won't be an exact binary representation of 9 bytes.

TUPLE cannot be expanded.

You can get the size of basic types using the PLATFORM class, or when using REFLECTOR, you can get a REFLECTED_OBJECT from any object and you can then query the physical size of that object which also contains all the runtime overhead necessary.

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
However, how can one declare a C-style string, e.g., something like a
char text[16];
SECOND QUESTION: Is there a way to declare something equivalent to a char [n]
where n is a fixed value?

There is no automatic way to declare such a type statically apart from encoding this into the name. But usually it is better to use STRING or ARRAY with the proper count. Retrieving data is never an issue, and storing it would have to verify the size constraint of 16 bytes.

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
I know that ARRAY maintains its data in a SPECIAL [G] attribute, but declaring a
SPECIAL [CHARACTER_8] element in the tuple above would not allocate 16 bytes.
One could create a TEXT_16 class whose creation routine would allocate the 16
bytes, but I think that still wouldn't help because what's really needed is an
expanded class.

Why do you need an expanded class? If you encode the information properly, you just need to allocate what you need. If you create your own format, then you could devise a scheme where when a STRING is expected, the length of the string is read and then you read everything up to that length, or you can choose the null terminated approach too (although not as safe with respect to security).

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
FINAL or COROLLARY QUESTION: Is there a way to do what I describe above purely
in Eiffel, or is it necessary to descend to a C interface?

You can do everything in Eiffel. No need to go down to C.

Regards,
Manu

[Joe Abbate [e-mail] [eiffel_software]]

Hello Manu,

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
Why do you need an expanded class? If you encode the information
properly, you just need to allocate what you need. If you create your
own format, then you could devise a scheme where when a STRING is
expected, the length of the string is read and then you read everything
up to that length, or you can choose the null terminated approach too
(although not as safe with respect to security).

I'm not creating my own format. The format is given by some external
aplication (like Larry Rix mentioned that he had some dBase DBF files).

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
You can do everything in Eiffel. No need to go down to C.

Let me posit the following C structure, could be an employee record with
id, name, salary or a product with id, name and price:

struct record { int id; char name[16]; float amt; } buf;

In C it's almost trivial to use fread(&buf, sizeof struct record, ...)
and fwrite to read from and write to a file containing such records.
OK, so I understand that in OO you want to deal with part of that
structure separately. That means in Eiffel, one would have to do
something like this (assuming f is a RAW_FILE):

f.read_integer_32
id := f.last_integer
f.read_stream (16)
name := f.last_string
f.read_real
amt := f.last_real

Given a more typical record with say 50 such fields, including dates,
conversions like 0/1 to BOOLEAN and whatnot, the number of statements
goes to at least double the number of fields, without considering some
supporting instructions to properly report any errors.

Is the above more or less correct or is there something to ease the
tedium of dealing with each field separately?

Joe

['Emmanuel Stapf' [e-mail] [eiffel_software]]

Let me posit the following C structure, could be an employee record with id,
struct record { int id; char name[16]; float amt; } buf;
In C it's almost trivial to use fread(&buf, sizeof struct record, ...) and
fwrite to read from and write to a file containing such records.

Although it works for one platform, using fread/fwrite is not portable. A C portable solution would have to read field by field and this is what Eiffel does.

OK, so I understand that in OO you want to deal with part of that structure
separately. That means in Eiffel, one would have to do something like this
f.read_integer_32
id := f.last_integer
f.read_stream (16)
name := f.last_string
f.read_real
amt := f.last_real
Given a more typical record with say 50 such fields, including dates,
conversions like 0/1 to BOOLEAN and whatnot, the number of statements goes to at
least double the number of fields, without considering some supporting
instructions to properly report any errors.

Yes but at least you can provide a very precise error to the user as you would know exactly where it is failing.

Is the above more or less correct or is there something to ease the tedium of
dealing with each field separately?

Currently I don't believe anyone has built an easy way to parse binary files however as I said in my previous email, I would not see why this would not be possible. If you do not have any fixed size arrays, then the TUPLE solution and a little bit of introspection would work well. Otherwise, one has to build a text representation of the binary representation and then it could produce a TUPLE object of the right type.

Something like:

r: READER

create r.make_with_specification ("i4, c1[16], f4")

r.read_entry

if attached {TUPLE [id: INTEGER: name: STRING; amt: REAL_32]} r.last_entry as l_entry then
-- Do something with `l_entry'
end

In the above, one has to write the READER class to parse the expected binary sequence, but then it can read any binary formats.

Manu

[Joe Abbate [e-mail] [eiffel_software]]

Hello Manu,

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
Although it works for one platform, using fread/fwrite is not portable.
A C portable solution would have to read field by field and this is what
Eiffel does.

Indeed, I've just been reminded that using fwrite on the plain structure
leaves gaps between fields (due to in-memory alignment), so it becomes
necessary to use dummy read_stream (n) calls to skip over those gaps.

I have now bumped into another problem. My approach to reading both a
PLAIN_TEXT_FILE and a RAW_FILE has been:

from f.open_read
until f.end_of_file
do -- read and process the fields
end

In the text case, using f.read_line I discovered that Eiffel would
attempt to read beyond the last newline. I bypassed the problem by
using if f.last_string.count > 1 then process. In the raw case, it just
so happens that the first field in my particular example is a single
character. Eiffel reads beyond EOF using an external C function
(eif_gib?) and in the debugger I see the value 255 has been read, i.e.,
a -1 (C EOF indicator). But strangely, Eiffel continues and attempts to
read the second field (a string) and only then detects the error
(because the file has to be is_readable--precondition in the read_x
routines). I haven't dug into this yet, but I haven't seen anything
that cautions one about this behaviour. It seems that either the caller
has to check is_readable before each field read_x or a rescue clause is
needed that has to differentiate between having read a full record (or
almost, because of the -1), or finding a truly truncated file. Either
way, it's disconcerting.

Joe

['Emmanuel Stapf' [e-mail] [eiffel_software]]

I would recommend to always use RAW_FILE instead of PLAIN_TEXT_FILE because the magic of converting the new line character on Windows is quite cumbersome. And that's maybe this is what you are hitting.

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
from f.open_read
until f.end_of_file
do -- read and process the fields
end

That pattern should work just fine, this is what we use in our code without problems. Note that `end_of_file' will only be set after reading something.

Regards,
Manu

[Joe Abbate [e-mail] [eiffel_software]]

Hello Manu,

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
I would recommend to always use RAW_FILE instead of PLAIN_TEXT_FILE
because the magic of converting the new line character on Windows is
quite cumbersome. And that's maybe this is what you are hitting.

No, I'm on Linux and I only used PLAIN_TEXT_FILE on an actual ASCII text
file with NL line-terminators.

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
from f.open_read
until f.end_of_file
do -- read and process the fields
end

That pattern should work just fine, this is what we use in our code
without problems. Note that `end_of_file' will only be set after reading
something.

If I'm reading you correctly (pun intended), then the code would have to be:

from f.open_read
until f.end_of_file
do
f.read_<something>
if not f.end_of_file then
-- process something
-- and read and process the rest
end
end

Joe

['Emmanuel Stapf' [e-mail] [eiffel_software]]

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
from f.open_read
until f.end_of_file
do
f.read_<something>
if not f.end_of_file then
-- process something
-- and read and process the rest
end
end

I better understand what kind of code you wrote and the test you had for the count being non-empty. The code above is correct but for the `read_stream' or `read_line' or `read_word' routine you might still have something read in `last_string'. So here is what one needs to understand:

f.read_line
if not f.end_of_file then
-- `last_string' has been updated and could be empty (case of an empty line)
else
-- `last_string' might not be empty (case of the last line of
-- a file that is not terminated by a new line).
end

Regards,
Manu

[Joe Abbate [e-mail] [eiffel_software]]

Hello Manu,

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
Currently I don't believe anyone has built an easy way to parse binary
files however as I said in my previous email, I would not see why this
would not be possible. If you do not have any fixed size arrays, then
the TUPLE solution and a little bit of introspection would work well.
Otherwise, one has to build a text representation of the binary
representation and then it could produce a TUPLE object of the right type.

I've managed to read the records properly without too much trouble. For
the strings, which happened to be null-terminated, I first use RAW_FILE
read_stream (<known max length of string + null>) and then used STRING
head (<string>.index_of ('%U', 1)) to trim it.

On the write/rewrite side, I'm having trouble figuring out what is
needed. I know I have to use RAW FILE put_managed_pointer (<some
managed_pointer>, 1, <known max length + null>) but there are confusing
options about how to transfer or point that MANAGED_POINTER to the data
in the STRING object. I know the data is in the SPECIAL area attribute
of STRING but haven't yet found any clear descriptions either in the
code or the documentation as to the route from the SPECIAL to the
MANAGED_POINTER: through C_STRING, POINTER or what, and how?

Joe

[Joe Abbate [e-mail] [eiffel_software]]

Hello again Manu,

Post by Joe Abbate ***@freedomcircle.com [eiffel_software]
Hello Manu,

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
Currently I don't believe anyone has built an easy way to parse binary
files however as I said in my previous email, I would not see why this
would not be possible. If you do not have any fixed size arrays, then
the TUPLE solution and a little bit of introspection would work well.
Otherwise, one has to build a text representation of the binary
representation and then it could produce a TUPLE object of the right type.

I've managed to read the records properly without too much trouble. For
the strings, which happened to be null-terminated, I first use RAW_FILE
read_stream (<known max length of string + null>) and then used STRING
head (<string>.index_of ('%U', 1)) to trim it.
On the write/rewrite side, I'm having trouble figuring out what is
needed. I know I have to use RAW FILE put_managed_pointer (<some
managed_pointer>, 1, <known max length + null>) but there are confusing
options about how to transfer or point that MANAGED_POINTER to the data
in the STRING object. I know the data is in the SPECIAL area attribute
of STRING but haven't yet found any clear descriptions either in the
code or the documentation as to the route from the SPECIAL to the
MANAGED_POINTER: through C_STRING, POINTER or what, and how?

Since I didn't get a reply, I tried various alternatives and finally
came up with the following scheme:

local
text, dummy: STRING
textp: MANAGED_POINTER
do
create text.make (<max size of STRING fields>)
create textp.share_from_pointer (text.area.base_address,
text.area.capacity)
dummy := ""

-- for each STRING field in the record
text.share (record.fieldN)
textp.set_from_pointer (record.fieldN.area.base_address,
record.fieldN.area.capacity)
raw_file.put_managed_pointer (textp, 0, <actual size of fieldN>)

-- for any needed alignment
text.share (dummy)
textp.set_from_pointer (textp, 0, <size of adjustment>)

This appears to work (although perhaps dummy needs to be created with
the max size for an adjustment), but it looks like a huge kludge that is
liable to break if some implementation detail is changed. So I would
still appreciate if you or anyone else has any pointers to
documentation, other writings or working code that explains or shows how
to do this properly. Thanks.

Joe

['Emmanuel Stapf' [e-mail] [eiffel_software]]

Since I didn't get a reply, I tried various alternatives and finally came up
local
text, dummy: STRING
textp: MANAGED_POINTER
do
create text.make (<max size of STRING fields>)
create textp.share_from_pointer (text.area.base_address,
text.area.capacity)
dummy := ""
-- for each STRING field in the record
text.share (record.fieldN)
textp.set_from_pointer (record.fieldN.area.base_address,
record.fieldN.area.capacity)
raw_file.put_managed_pointer (textp, 0, <actual size of fieldN>)
-- for any needed alignment
text.share (dummy)
textp.set_from_pointer (textp, 0, <size of adjustment>)
This appears to work (although perhaps dummy needs to be created with the max
size for an adjustment), but it looks like a huge kludge that is liable to break
if some implementation detail is changed. So I would still appreciate if you or
anyone else has any pointers to documentation, other writings or working code
that explains or shows how to do this properly. Thanks.

Clearly this code is very unsafe because of the use of pointers. So I would not recommend it.

Looking at your code, I have a few remarks. It is not clear why you call `text.share' and then never use `text'. Also we are talking bytes here so the actual size of the fieldN is the same as `fieldN.count', isn't it?

In this case, I think you just have to write:

raw_file.put_string (record.fieldN)

After that you seem to have some padding, so you can simply repeat

raw_file.put_character (padding_character)

as many times as required.

Am I missing something here?

Regards,
Manu

[Joe Abbate [e-mail] [eiffel_software]]

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
Clearly this code is very unsafe because of the use of pointers. So I
would not recommend it.
Looking at your code, I have a few remarks. It is not clear why you call
`text.share' and then never use `text'. Also we are talking bytes here
so the actual size of the fieldN is the same as `fieldN.count', isn't it?

It was mostly a process of trial and error and thus suffered from
previous attempts. From what I determined, to use put_managed_pointer
the string has to be shared with the pointer. I was hitting a
not_shared precondition violation otherwise.

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
raw_file.put_string (record.fieldN)

Doesn't that use the length of the string in the current object, i.e.,
if record.fieldN has a value of "Manu", doesn't it write four bytes,
even though the area has allocated say 16 bytes for the name?

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
After that you seem to have some padding, so you can simply repeat
raw_file.put_character (padding_character)
as many times as required.

In my example, the padding is fixed, because the whole record structure
is fixed, and the padding is merely due to record alignment in memory,
i.e., if the record has a 32-bit integer, a single character and a
float, in that order, the padding between the single character and the
float will typically be three bytes. If the put_string routine writes
only the STRING count, then an additional padding would have to occur in
a loop that wrote the padding character <capacity - count> times, and
then the alignment padding if needed.

Joe

['Emmanuel Stapf' [e-mail] [eiffel_software]]

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
raw_file.put_string (record.fieldN)

Doesn't that use the length of the string in the current object, i.e., if
record.fieldN has a value of "Manu", doesn't it write four bytes, even though
the area has allocated say 16 bytes for the name?

That's correct.

Post by 'Emmanuel Stapf' ***@eiffel.com [eiffel_software]
After that you seem to have some padding, so you can simply repeat
raw_file.put_character (padding_character)
as many times as required.

In my example, the padding is fixed, because the whole record structure is fixed,
and the padding is merely due to record alignment in memory, i.e., if the record
has a 32-bit integer, a single character and a float, in that order, the padding
between the single character and the float will typically be three bytes. If
the put_string routine writes only the STRING count, then an additional padding
would have to occur in a loop that wrote the padding character <capacity -
count> times, and then the alignment padding if needed.

Indeed. So I thing using put_string and put_character with a padding character is a much safe solution.

Manu