Category : Science and Education
Archive   : HDFNEWS.ZIP
Filename : HDFNEWS1.TXT

 
Output of file : HDFNEWS1.TXT contained in archive : HDFNEWS.ZIP

NCSA HDF Newsletter #1
December 12, 1989

Contents:
The HDF Newsletter
The Current State of HDF
HDF Release 3.0
The HDF Vgroup: A More General HDF Structure
HDF File Conversion
The NCSA HDF Staff


--------------------------- ** ---------------------------

The HDF Newsletter

As HDF has begun to spread beyond NCSA many of you have requested some form of
regular communication among all HDF users. The HDF Newsletter is an
occasional newsletter designed to provide that communication. It is meant to
serve as a clearinghouse of information about HDF.

The contents of this issue reflect the kinds of things I expect to include in
the Newsletter, but I am quite open to suggestions. I will print anything
about HDF that seems as though it might be of interest to the HDF community.

We will distribute the Newsletter via both surface mail and email. If we have
your email address, we will use email. It is cheaper and (we hope) easier to
manage than surface mail.

Mike Folk
NCSA
University of Illinois
605 E. Springfield Ave.
Champaign, IL 61820
217-244-0647
[email protected] (internet)
12409@ncsavmsa

--------------------------- ** ---------------------------

The current state of HDF

FUNCTIONALITY
HDF Release 2.0, released last February and since then revised only for bug
fixes, is still the most up-to-date official version of HDF. It includes
interfaces and file formats for handling 8- bit raster images (with or
withoout palettes), and regular gridded multidimensional floating point data
sets. It includes utilities for listing the contents of an HDF file (hdfls),
for converting raw raster data to HDF (r8tohdf) and vice versa (hdftor8), and
for displaying images or sequences of images from HDF files (hdfseq and
hdfrseq).

NCSA WORKSTATION TOOLS
Most of the workstation tools now support HDF files. This includes
CompositeTool on the Sun, which previously did not support HDF. (By the end
of the year ImageTool (also Sun) will also support HDF files.) The Mac tools
are NCSA Image, NCSA PalEdit, NCSA DataScope, and NCSA Layout. NCSA
CompositeTool is a Sun version of Layout. Our two X-Windows tools are NCSA
X-Image, and NCSA X-DataSlice.

OFFICIAL PLATFORMS
Architectures that we have HDF running on: Cray-2 and Cray X-MP; Alliant
(Concentrix), SGI 4, Sun-3, and Sun Sparc machines, Macintosh and IBM-PC. And
VMS, sort of (see below).

UNOFFICIAL PLATFORMS
People outside of NCSA have ported HDF to: Convex, Stellar, Connection
Machine, Symbolics Machine, and Amiga. (Do you know of any others?) I think
most of these people would be happy to give you advice on any special thing
you need to do if you want to port HDF to one of these machines, but I
probably shouldn't publish their names without their permission. Let me know
if you are interested in any particular port, and I will give your name to the
relevant person.

SEMI-OFFICIAL PLATFORMS -- VMS & CTSS
VMS. A lot of you have asked for HDF support on VMS systems, so we made a
quick attempt to port HDF to VMS last spring. It was problematic from the
beginning because we don't have any real VMS expertise here. It actually went
reasonably well in most respects, except for one major problem, which we're
still struggling with: VMS-C writes files "stream-LF" format, which does not
transfer well via ftp and some other transfer programs. There is a VMS
utility called FIXATR that we supply with VMS HDF, which converts stream-LF
files to "fixed-512" files, which seems to be more amenable to transfer
outside of the VMS environment. This is not a great solution, since it
involves an extra step whenever files go to or from VMS, but it may at least
make it possible. Any suggestions for better solutions are certainly welcome.

CTSS. CTSS is a different story. Not only don't we have CTSS/HDF expertise at
NCSA; we also don't have access to CTSS any more. If you are having problems
getting HDF to work on CTSS, we will be happy to try to help you, but frankly
we haven't been very successful diagnosing such problems recently. If we
can't help you, we may be able to put you in touch with someone who has
successfully implemented HDF on CTSS.


--------------------------- ** ---------------------------


HDF Release 3.0

After many delays, HDF 3.0 seems now to be ready for release. All of the
features of HDF 2.0 are present in HDF 3.0, plus several new features.

HDF 3.0 has been in available as a beta release for about 5 months, and it
seems pretty bug free. But certain parts of it have hardly been used by
anyone, so please let us know if something doesn't seem to work the way you
think it should.

New features of HDF 3.0 include the following:

An interface for basic i/o of 24-bit raster images, which includes the
following routines:

DF24setil: sets the interlace to be used when writing out the RIS24 for
the image.

DF24addimage: appends a 24-bit raster image set to the file.

DF24getdims: retrieves the dimensions and interlace of the image.

DF24getimage: retrieves the image and stores it in an array.

DF24reqil: specifies an interlace to be used in place of the interlace
indicated in the file when the next raster image is read.


An interface for annotating HDF data objects and files, which includes the
following routines:

DFANgetlablen: gets length of label of a tag/ref

DFANgetlabel: gets label of tag/ref

DFANgetdesclen: gets length of description of tag/ref

DFANgetdesc: gets description of tag/ref

DFANputlabel: puts label of tag/ref

DFANputdesc: puts description of tag/ref

DFANlastref: returns ref of last annotation read or written

DFANlablist: gets list of labels for a particular tag


An interface for input and output of 8-bit palettes, including the following
routines:

DFPaddpal: appends a palette to a file.

DFPgetpal: reads in the next palette in the file.

DFPputpal: writes a palette to a file.

DFPnpals: indicates number of palettes in a file.

DFPwriteref: sets the reference number of the next palette to be written.

DFPreadref: gets the reference number of the next palette to be retrieved.

DFPrestart: specifies that the next call to DFPgetpal reads first palette
in the file, rather than the next.

DFPlastref: returns value of the reference number most recently read or
written.


Scientific data set routines for storing and retreiving subsets (slices) of
scientific data, and for choosing optional storage formats and data types:

DFSDstartslice: prepares to write part of dataset to file.

DFSDputslice: writes part of a dataset to a file.

DFSDendslice: indicates write completion for part of dataset.

DFSDgetslice: reads part of a dataset.

DFSDsettype: specifies data attributes: data type and representation,
system type, and array order.


* new utilities, including the following:

hdfed: lets you browse in an HDF file and manipulate some of the data

fptohdf: converts floating point data to HDF floating point data and/or
8-bit raster images

r24tohdf: converts a raw RGB 24-bit image to an 8-bit RIS8 with a palette

paltohdf: converts a raw palette to hdf format

hdftopal: converts palette in an hdf file to raw format

--------------------------- ** ---------------------------


HDF Vgroup--A More General Structure

HDF currently supports only two major types of scientific data: raster data
and regular gridded multidimensional arrays. Recently we have added an HDF
structure that promises to expand significantly the types of data that we can
support. This structure, currently called Vgroup (the name may change),
provides two important new structures:

1. a general grouping structure that lets the user form groups out of any
set of HDF objects, including other Vgroups

2. a general structure made up of a set of record-like structures, each
record being made up of a set of fields. Fields can be use-defined or
predefined.

Vgroups look very promising for a number of important scientific application
areas not currently supported by HDF, including finite element and
non-rectilinear mesh data. We have talked with a number of scientists who
work with this kind of data, and our general impression is that there is a
need for a standard in this area and that Vgroups could well provide the
standard.

The idea for Vgroup springs from a need to store 3-D polygonal data, with
vertices, polygons (connectivity lists), and various associated values with
each vertex or polygon.

When Jason Ng took over the Vgroup project, he began talking to a lot of
potential users from many different disciplines about how they might be able
to use Vroups. Their responses were so varied, that Jason immediately began
looking for ways to generalize the concept so that it could handle many
different kinds of data. The result is a very general HDF structure that
"groups" one or more other HDF structures. The structures in a Vgroup can be
anything you want them to be including other Vgroups.

For example, a Raster Image Set could probably be stored as a Vgroup. The
members of the Vgroup would be a palette, a dimension record, and an image.
But with the Vgoup concept we could now go a step further and group several
Raster Image Sets, in an animation, for example.

While the Vgroup idea provides a general structure for linking HDF items
together, we still need a structure for representing things like sets of
vertices and connectivity lists. The structure that we use for this is a very
familiar one--a field and record structure. Store 3-D vertices, we define
three fields per element, corresponding to the x, y and z coordinates that
define each vertex. A vertex set is a fixed number of vertex records. A
polygon set is similarly defined. If there are four vertices per polygon,
each record consists of four vertex numbers; these numbers appear an order
that describes the connectivity of the polygon.

In keeping with our desire to standardize those items that are likely to be
accessed by different programs in different environments, certain types of
sets will be predefined. A 3-D vertex set will have exactly three fields per
vertex, for instance. But those who have the need are free to define their
own dataset types. For example, you might for some reason want to store
scalar values in the same dataset that you store your vertices. You are free
to do this, but must recognize that you are building a non-standard dataset.
(Unless, of course, enough users ask us to make THAT one of the standard
types.)

There are still some issues yet to be settled with respect to Vgroups, but we
think that we are pretty close to having the major design of it pinned down.
The interface is now undergoing a major overhaul. We expect to release a Beta
version of it in mid- January for any of you who would like to look at it and
play with it.

Of course we welcome all comments and questions you have about Vgroups. We
don't want to freeze this structure too soon, because we see it as an
important building block to HDF in the future. On the other hand, we want to
get it into use as soon as is reasonably possible. If don't want to wait for
the Beta release, contact us and we will send you the draft of the
documentation.

--------------------------- ** ---------------------------


HDF File Conversions

A frequent question that arises is "How can I translate between file format
xxx and HDF?" We want very much to support translators between HDF and other
formats, but have so far had trouble finding the resources to write them.
Here is a list of some of the translators that we would like to have. If you
have a translator, know of one, are interested in working on one, etc., please
let us know.


FITS--Flexible Image Transport System
FITS is the standard format used for astronomical images and other digital
arrays. We have small collaborative project with the Space Telescope Science
Institute to translate basic FITS to HDF. We hope this will lead to a more
elaborate project later.

CGM--Computer Graphics Metafile
CGM is a very widespread file format that is used primarily for describing
pictures. Though CGM and HDF have different roles to play in scientific
visualization, it would be nice to be able to look at CGM pictures using
HDF-based tools, and vice versa. Some programs that might help us do this: a
CGM cell array-to-HDF converter, a rasterizer that converts CGM 2-D pictures
to HDF raster, and a converter that converts CGM text to HDF. (HDF currently
does not support text.)

(We have heard about a tool called CGM-Maker developed at Los Alamos that
converts between CGM and Pict files, among others. Since NCSA Layout reads and
writes both Pict and HDF files, CGM- Maker and Layout together provide a kind
of primitive filter between the two formats.)

netCDF
The netCDF interface allows users to share scientific data in a form that is
self-describing and network transparent, and is very much in the spirit of
HDF. It is a well-designed, flexible interface, and one that would benefit
HDF users enormously if it could be incorporated into the HDF library. We are
very interested in adapting HDF to support the netCDF interface, and also in
writing translators that convert between the HDF and netCDF file formats.

TIFF--Tag Image File Format
Several HDF users have requested translators to and from this common image
format and HDF.

--------------------------- ** ---------------------------

The HDF Staff

In the last year and a half, the HDF project has expanded from one programmer
to six people. We lost Swami Natarajan, who finished his Ph.D last summer and
took a job a Texas A & M. We really miss him and continue to appreciate the
work he did for us. Fortunately he is still in touch via email.

Mike Folk is the project manager for HDF. Mike joined NCSA in August, 1988,
having last worked at Oklahoma State University as a professor in the computer
science department.

ChinChau Low has taken over Swami's responsibilities for maintaining HDF.
ChinChau is a graduate student in Computer Science at the University of
Illinois. ChinChau joined us in Fall, 1988, and has worked on virtually all
aspects of HDF. He is crucial to maintaining HDF and also to all additions.

Jason Likkai Ng is a full-time staff member from Malaysia (via Cornell,
Milwaukee, and La Jolla) who joined us in May. Jason's main responsibility is
the Vgroup project described earlier.

Peter Webb, Brian Calvert and Drew Hess joined the HDF group this fall
semester. Peter last worked at Schlumberger; Brian joins us from Motorola.
Peter and Brian are graduate students in Computer Science; Drew is an
undergraduate in Computer Science.

Peter's current projects include (1) installation of a revision control system
to help manage HDF code development, and (2) finding ways to speed up HDF's
performance. Brian's project is Polyview, an SGI-based interactive tool for
displaying polygonal data stored in the Vgroup format described above. Drew
is currently working on an HDF utility for taking slices out of 3-D scientific
datasets.



  3 Responses to “Category : Science and Education
Archive   : HDFNEWS.ZIP
Filename : HDFNEWS1.TXT

  1. Very nice! Thank you for this wonderful archive. I wonder why I found it only now. Long live the BBS file archives!

  2. This is so awesome! 😀 I’d be cool if you could download an entire archive of this at once, though.

  3. But one thing that puzzles me is the “mtswslnkmcjklsdlsbdmMICROSOFT” string. There is an article about it here. It is definitely worth a read: http://www.os2museum.com/wp/mtswslnk/