Sonar Images as netcdf files - status and history E. Montgomery 8/04/09 update- BIG CHANGES from before 1)The method of providing metadata to the conversion programs has been changed to follow how ADCP, ADV, and PCADP processing is handled. The programs to use for all processing are called fanmeta_example.m, penmeta_example.m or (soon) azmmeta_example.m- there still has to be one file per type of sonar, since the parameters are different. Some of the changes include: - all outputs are .cdf files - the mfile does all steps of processing, and includes the metadata - uses read_global_atts() to get the same metadata as other instruments - uses structures to pass arguments to processing programs - uses if 0/1 blocks to enable procecessing steps - can do mkrawcdf and do_fan_rots (or do_pen_proc for pencil) in one run 2) For Hatteras09 a file that contains the cartesian measurements of where things were installed on the tripod, plots the result and returns a structure contianing the measurements was created. These are called orient###meta.m, rotfc_### uses orient###meta output to overplot the tripod and it's features on a fan image. 3) new templates for making animations exist- sfanvm.m, spenmv.m. These still use matlab functions to make the movie- not saving individual frames to combine in other software. 4) there are two ways to plot azimuth data in the works: plotrange_cdf.m (in sonar_lib) and plot_azm.m (in branches/em_branch/sonar). Both are works in progress, so will continue to evolve. Plotrange_cdf was written by MM and EM and gives the user various views of the azimuth data. If you choose plottype='2d' you get to see the raw sweeps, rotation by rotation. If you choose '3d' you get an interpolated version of the profile_range data (sparse and smoothed). If you choose 'scat_frm_image' you get a scatter plot of the data at each point along each scan from the raw_image. Finally '3d_frm_img' interpolates raw data from the image to a regular grid. This works if pencil_tilt ==0- other pencil tilts lead to unexpected results. Plot_az.m was written by CRS and uses a routine called gridfit (in m_contrib/trunk/gridfitdir). It works on the older (MVCO_05) files where the X and Y coordinates are reversed, and requires hand editing to change source location. Plot_azm.m is EM's version of plot_azm where the newer files are read, and can be passed as an argument. This version still trys to blank the feet based on an algorythm that worked for MVCO_07, but doesn't for Hatteras-09, and it's unclear that the pencil_tilt is applied as it should be. This program also seems to filter out the bar that's obvious in the images made by plotrange_cdf, so has some hidden stuff going on. <<< In 2008, there was a request to separate procsonar07 into smaller components. One program, mkrawcdf.m makes the .cdf files for Pencil, fan and azimuth data. Do_fan_rots handles rotating the fan data, and interpolating onto a north is up circular plot. Do_pen_proc converts the raw scans, rotates for transducer tilt angle and makes plots of the seabed under the transducer where up is up. The 2008 software evolution isn't fully documented. >>> 11/23/07 update procsonar07 on cmgsoft svn processes fan, pencil, and azimuth pencil images. It outputs netcdf format files for all three types of data. For each type of sonar data, there will be *_raw.cdf that contains the data as it comes out of the instrument and *_proc.cdf that has images of the sweeps having done the trig and interpolation to make something sensible in a "earth" view. The azimuth files are too big to save a whole deployment in one file, so I'm planning to change procsonar so azimuth files are stored by day instead of arbitrary groups of 25 samples (or experiment for fan and pencil). readrangeall.m and plotrange.m were added to read and plot the azimuth files (.R##). Plotrange with 3draw is reasonably good, buth with sparse data like from MVCO07, we need better plotting options. There's a wrap at 0 or 180 that shouldn't be there, so needs attention. animsonar_nc. does what animsonar used to do but reads the netcdf files currently output. Use anim_meta07 as input. 6/21/07 update The pencil beam .cdf files are now also ready for evaluation. Pen_0314n_raw and _proc are both also in "Temporary Items". These are also missing units. The other thing that is odd is that neither pcolor nor imagesc works correctly with autonan_on. autonan_on ncload pen_0314n_proc.cdf; imagesc(squeeze(sonar)image(6,:,:)); shading flat works fine but when you try to get the x and z axes on there correctly, the autonan doesn't work right on x. plot(x) correctly shows just the progression from -2 -> 2; doesn't show 1e35s imagesc(x,-z,squeeze(sonar)image(6,:,:)); plots a shape indicating data / no data imagesc(x(250:540),-z, squeeze(sonar)image(6,:,:)) plots what we expect to see I'm not sure why this is, but the data is there, it just doesn't plot that nicely. Probably related to whether xdst(ia)=xint; or xdst=xx; (lets x hold the whole -5 to 5 array) xdst(ia)=xint should be right for the 03/14 data, since the range changed during the day. For our more normal data, Range won't change, so the whole ia thing may be unnecessary. original notes: 6/12/07 I think we're close to having the fan beam data into an acceptable netcdf format. At the moment, Procsonar07 calls readfan and plotfan07, instead of showfan07, which used to do both reading and plotting. Readfan.m reads and saves all the Imagenex header stuff into structures (a great improvement). Plotfan07.m (there was an eariler plotfan I didn't want to trash) does the conversion to polar; does the rotation for declination, adcp angle offset, and math->map convention, then plots. Procsonar07 takes the header elements and either saves them as attributes or combines individual values (1/scan) into a vector (n scans) to be stored as another variable with the image. It writes 2 netcdf files, one "_raw", which includes head angle, attributes and the unrotated image data; the other, "_proc" contains the polar view, rotated such that N is up (+y), and the x, and y vectors that the polar view was interpolated onto. There's enough info in the attributes that the user can discern what the values for each rotation were. With regard to data flow, my intention was that the user would do a best guess of inputs on the first run of procsonar07, then if re-doing the rotations was required, the _raw file could be read and different rotation parameters could be substituted prior to re-running plotfan07. I put f0314_raw.cdf and f0134_proc.cdf on capecodder in "Temporary Items" (which is at the same level as "data") for people to look at. The only missing info I'm currently aware of are the units for both sonar images. I'll fill these in once I know what they are. For keeping track of the orientation and alignment data for the tripod, I suggest we include an attribute that points to a file containing all the orientation info for the tripod. That way if we get something wrong, it can be corrected independently from the .cdf with the images. We could include "nc.TripodAlignmentFile = ori_name.txt" in all the instrument files for each tripod. The orientation file might look something like tripod_defs.m (also on capecodder in "Temporary Items" The netcdf for Pencil is not yet ready for primetime. I had something that worked OK for the single sweep, using the .mat output, but will need to modify it when a readpencil.m is written (similar to readfan, that outputs the header info as structures), and after we get some radial sweeps to play with. 25 May. 07 4/25 + Marinna wrote a generic reader (readfan.m) for the fanbeam, that'll work with data from our logger and that logged by the Imagenex viewer. + readfan puts imagedata into a matrix with the rows and columns reversed from how showfan reads it. + ProcsonarMM was made from procsonar_apr07, and allows the user to comment out showfan07 and use readfan and plotfan07 to read and then plot the image or vice versa for verification. + Readfan and plotfan07 use all information in the header, instead of user entries in the metadata.txt. + plotfan07 takes the inputs in reversed row-column data- I didn't want to rot90 & fliplr arbitrarily (plotfan is something older- I don't know what it was for, but kept it, in case it was important.) + plotfan07 was validated against the 031407 test data with the tray to the right of the blank zone and using CRS particle tripod geometry on an MVCO image. History- Animatesonar.m, animsonar1023.m were versions of the intital program- it calls showfan.m and showpencil.m to make the plots. Written by Marinna, Chris, and Charlene. Features: raw data conversion for Fan and Pencil (fan rotation was not right and pencil wasn't right either) plot making with overlays of currents and waves movie making netcdf output (didn't work) 4/13/07 After working with animsonar1027, etm decided to separate the sonar processing functions from making the combined plots and movie making Now, procsonar_apr07 is the program that converts the raw sonar files into a) netCDF and b) .mat structures, and amimsonar_apr07 is used only to make the overlay plots and generate movies from them and the adcp data. procsonar_apr07 uses showfan07.m and showpen07.m showfan07 was validated using data from the March 13 dock test showpen07 uses Doug Wilson's was of plotting the pencil data, and validated using 3/14 data procsonar_apr07 must be run twice- once for each sonar. To convert each sonar the user must create a metafile, for which the fan and pencil values are entered. See fan0314meta.txt and pen0314meta.txt. procsonar_apr07 makes a netcdf files and .mat files that contain the image data. The netcdf files aren't quite right now. to do: correct netcdf output of the data combine the fan and pencil specific meta values into one file for procsonar07 input get the plt_tripod_tst code to work so overlays are aligned make the combined plotting and movie making work Note on standard data to use in validation: March 14 dock test: (See rep_dt_0314.doc for details) orientation: a tray was suspended along the pencil path. offset to the right, as looking from the fan dead zone and pencil red transducer end (both are nominally aligned). There was no adcp data, but the adv ran, so the adcp_beam3 should be the adv's North. Shapes: three bedforms were put into the sand in the tray: flat 50 cm long 8cm high waves 10 cm long 4cm high ripples.