4/17/08 Moved the loop that processes each sweep(s) to do_fan_rots from procfan08. Now it can write to the cdf file sample by sample. The new 8369fan0416_hrproc.cdf on capecodder has all the samples, both sweeps at dx=.01. sfanmovie, as modified by RPS can make a .avi movie of the desired scans. 4/11/08 In reprocessing with dxy=.01 instead of .02 I ran into memory issues storing the Zs array before writing it to the _proc.cdf. By only doing the 50 scans that would go in the animation (132:181) it would *just* go. sfanmovie.m made the current version from 8369fan0411_hrproc.cdf to create fan_run0411.avi, which are both on capecodder in MVCO_07\836sonar\outputs. 4/8/08 The updated files are on capecodder's stg/data/MVCO_07/836sonar/outputs. mkrawcdf_08 and defineRawSonarNCFile, were modified to work with Pencil and azimuth data, and create the smaller output files. The raw Azimuth data is still separated into 1-day segments. Fan processing is accomplished by running do_fan_rots.m, which calls procfan08 to do the rotation and interpolation. An output_proc.cdf is created. This file is read by sfanmovie.m to produce an animation. Sfanmovie == SIMPLE fan_movie. It donly does the fan images, no current or wave overlays. It keeps the secons axis where wave height would go, and marks time progress there. 3/21/08 At CRS request, created a way to process just the raw data, then use the raw.cdf file as input to the interpolation and making polar stuff. Part one is done- mkrawcdf_08.m creates just the _raw.cdf file. He also wanted smaller size, so converted some variables to short, and made others single dimension. (defineRawSonarNcFile.m). README for current state of the Sonar processing - 11/23/07 update procsonar07 on cmgsoft svn processes fan, pencil, and azimuth pencil images. It outputs netcdf format files for all three types of data. For each type of sonar data, there will be *_raw.cdf that contains the data as it comes out of the instrument and *_proc.cdf that has images of the sweeps having done the trig and interpolation to make something sensible in a "earth" view. The azimuth files are too big to save a whole deployment in one file, so I'm planning to change procsonar so azimuth files are stored by day instead of arbitrary groups of 25 samples (or experiment for fan and pencil). readrangeall.m and plotrange.m were added to read and plot the azimuth files (.R##). Plotrange with 3draw is reasonably good, buth with sparse data like from MVCO07, we need better plotting options. There's a wrap at 0 or 180 that shouldn't be there, so needs attention. animsonar_nc. does what animsonar used to do but reads the netcdf files currently output. Use anim_meta07 as input. 7/24/07 etm There are three sonar types : fan, pencil and range (rotating Pencil). Each has a program to read the data file and one to make diagnostic plots. readfan.m readpencil.m readrangeall.m plotfan07.m plotpen07.m plotrange.m The {read*, plot*} combinations above replace showfan07.m and showpen07.m, although they're still in this repository. The processing of fan and pencil sonar files may be accomplished using procsonar_07.m. This program processes multiple files in a directory (calling the above programs) and it uses the typical meta.txt file to input control parameters and descptive metadata. (See fanExamplemeta.txt and penExamplemeta.txt) Procsonar_07 outputs netcdf files of the raw data, and processed data, where the elevation and horizontal ranges are computed from the images. The programs that create the x-y coordinate elevations from the images are embedded in plotpen07.m At this point for data from the Azimuth drive, showsonars.m will call readrangeall and plotrange to display data from the pencil/azimuth combo. Animsonar_jun07.m is under development to do the next step of animating the timeseries and integrating data from ADCP, fan and pencil beams. **It isn't operational at this point. --------------------------------------------------------------------------------------- To work with image data from the pencil beam, some additional helper files stored under m_cmg/em_branches/sonar may be of interest. They may need adjustments to factors applied, depencing on the source of the data. find_good_pr.m : evaluates the % return of PenHeader.ProfileRange by 20 degree PenHeader.HeadAngle bins view_pen.m : Originally written for files made by the imagenex viewer, where there will be multiple scans in each file, possibly with gain changing too. Can be run on files from the logger, but gain returned will be []. Assumes you've run readpencil first: outputs matrices of ProfileRange, HeadAngle and gain. plt_prga.m : generates plots from the output of view_pen. plt_prfrngproc.m : another program to generate plots with ProfileRange overlaying the Pencil image. plt_17r02.m : view_pen for data from the azimuth drive, where the image is saved. Generates a plot of each sweep with ProfileRange overlaid. A .mat file with all the images is saved to facilitate reviewing the data. 6/21/07 update The pencil beam .cdf files are now also ready for evaluation. Pen_0314n_raw and _proc are both also in "Temporary Items". These are also missing units. The other thing that is odd is that neither pcolor nor imagesc works correctly with autonan_on. autonan_on ncload pen_0314n_proc.cdf; imagesc(squeeze(sonar)image(6,:,:)); shading flat works fine but when you try to get the x and z axes on there correctly, the autonan doesn't work right on x. plot(x) correctly shows just the progression from -2 -> 2; doesn't show 1e35s imagesc(x,-z,squeeze(sonar)image(6,:,:)); plots a shape indicating data / no data imagesc(x(250:540),-z, squeeze(sonar)image(6,:,:)) plots what we expect to see I'm not sure why this is, but the data is there, it just doesn't plot that nicely. Probably related to whether xdst(ia)=xint; or xdst=xx; (lets x hold the whole -5 to 5 array) xdst(ia)=xint should be right for the 03/14 data, since the range changed during the day. For our more normal data, Range won't change, so the whole ia thing may be unnecessary. original notes: 6/12/07 I think we're close to having the fan beam data into an acceptable netcdf format. At the moment, Procsonar07 calls readfan and plotfan07, instead of showfan07, which used to do both reading and plotting. Readfan.m reads and saves all the Imagenex header stuff into structures (a great improvement). Plotfan07.m (there was an eariler plotfan I didn't want to trash) does the conversion to polar; does the rotation for declination, adcp angle offset, and math->map convention, then plots. Procsonar07 takes the header elements and either saves them as attributes or combines individual values (1/scan) into a vector (n scans) to be stored as another variable with the image. It writes 2 netcdf files, one "_raw", which includes head angle, attributes and the unrotated image data; the other, "_proc" contains the polar view, rotated such that N is up (+y), and the x, and y vectors that the polar view was interpolated onto. There's enough info in the attributes that the user can discern what the values for each rotation were. With regard to data flow, my intention was that the user would do a best guess of inputs on the first run of procsonar07, then if re-doing the rotations was required, the _raw file could be read and different rotation parameters could be substituted prior to re-running plotfan07. I put f0314_raw.cdf and f0134_proc.cdf on capecodder in "Temporary Items" (which is at the same level as "data") for people to look at. The only missing info I'm currently aware of are the units for both sonar images. I'll fill these in once I know what they are. For keeping track of the orientation and alignment data for the tripod, I suggest we include an attribute that points to a file containing all the orientation info for the tripod. That way if we get something wrong, it can be corrected independently from the .cdf with the images. We could include "nc.TripodAlignmentFile = ori_name.txt" in all the instrument files for each tripod. The orientation file might look something like tripod_defs.m (also on capecodder in "Temporary Items" The netcdf for Pencil is not yet ready for primetime. I had something that worked OK for the single sweep, using the .mat output, but will need to modify it when a readpencil.m is written (similar to readfan, that outputs the header info as structures), and after we get some radial sweeps to play with. 25 May. 07 4/25 + Marinna wrote a generic reader (readfan.m) for the fanbeam, that'll work with data from our logger and that logged by the Imagenex viewer. + readfan puts imagedata into a matrix with the rows and columns reversed from how showfan reads it. + ProcsonarMM was made from procsonar_apr07, and allows the user to comment out showfan07 and use readfan and plotfan07 to read and then plot the image or vice versa for verification. + Readfan and plotfan07 use all information in the header, instead of user entries in the metadata.txt. + plotfan07 takes the inputs in reversed row-column data- I didn't want to rot90 & fliplr arbitrarily (plotfan is something older- I don't know what it was for, but kept it, in case it was important.) + plotfan07 was validated against the 031407 test data with the tray to the right of the blank zone and using CRS particle tripod geometry on an MVCO image. History- Animatesonar.m, animsonar1023.m were versions of the intital program- it calls showfan.m and showpencil.m to make the plots. Written by Marinna, Chris, and Charlene. Features: raw data conversion for Fan and Pencil (fan rotation was not right and pencil wasn't right either) plot making with overlays of currents and waves movie making netcdf output (didn't work) 4/13/07 After working with animsonar1027, etm decided to separate the sonar processing functions from making the combined plots and movie making Now, procsonar_apr07 is the program that converts the raw sonar files into a) netCDF and b) .mat structures, and amimsonar_apr07 is used only to make the overlay plots and generate movies from them and the adcp data. procsonar_apr07 uses showfan07.m and showpen07.m showfan07 was validated using data from the March 13 dock test showpen07 uses Doug Wilson's was of plotting the pencil data, and validated using 3/14 data procsonar_apr07 must be run twice- once for each sonar. To convert each sonar the user must create a metafile, for which the fan and pencil values are entered. See fan0314meta.txt and pen0314meta.txt. procsonar_apr07 makes a netcdf files and .mat files that contain the image data. The netcdf files aren't quite right now. to do: correct netcdf output of the data combine the fan and pencil specific meta values into one file for procsonar07 input get the plt_tripod_tst code to work so overlays are aligned make the combined plotting and movie making work Note on standard data to use in validation: March 14 dock test: (See rep_dt_0314.doc for details) orientation: a tray was suspended along the pencil path. offset to the right, as looking from the fan dead zone and pencil red transducer end (both are nominally aligned). There was no adcp data, but the adv ran, so the adcp_beam3 should be the adv's North. Shapes: three bedforms were put into the sand in the tray: flat 50 cm long 8cm high waves 10 cm long 4cm high ripples.