Table of Contents
List of Figures
List of Tables
Table of Contents
HIFI data are uniquely labeled by an obsid and, for now, are stored in HIFI ICC Versant Databases. Check the ICC Database contents listing to find which database an obsid is associated with, as well as for information about the observation. You will also find there the schema version of the database: you can only access databases that are at the same schema version as your HCSS software.
Databases are named according to the data's test enviroment, for example IST data is held in databases
labeled ist_fm_x_prop (x is an integer), while simulator data is stored in ds1. You can change database
within HIPE and JIDE but you can also set your preferred default in the property file (myconfig
for Linux) as follows:
hcss.binstruct.mib.source = pal hcss.binstruct.factory = herschel.hifi.share.binstruct.HifiBinstructFactory hcss.binstruct.services = herschel.binstruct.mib.MibPalServices
You define the database to be used with var.database.devel, for example:
var.database.devel = ilt_fm_5_prop${var.database.server} 0 READ
The hifi-cal database contains calibration information needed, for example,
to process data through the pipeline. There are two ways of getting this information into a Herschel
DP session. One is to use the Versant databases at the HIFI ICC, to do this you should add this line to
your config file:
hcss.binstruct.mib.pal.database = hifi-cal_dev@iccdb.sron.rug.nl 0 READ
The hifi-cal_dev and ilt_fm_5_prop_dev databases are always at
the latest schema version.
The other is to install the hifi-cal locally on your machine from a
copy of that at the ICC. First, download the local store copy of the
hifi-cal.
and unpack this in your .hcss directory. Second, change a Binstruct property so that
it will look for a local copy of the database:
#hcss.binstruct.factory = herschel.hifi.share.binstruct.HifiBinstructFactory hcss.binstruct.factory = herschel.binstruct.DefaultBinstructFactory hcss.binstruct.mib.pal.poolname = hifi-cal
This will now point to a local pool to get the MIB (Mission Information Base) needed for the HifiTimelineProduct.
hifi-cal_dev but, if you are using
a local pool, you will have to update it yourself. Do this by reloading the hifi-cal from the hifi-icc
pages or you might make a copy yourself by stepping through the copy-hifi-cal-to-local.py, which
you can find in {HIFI_HOME}/lib/herschel/hifi/scripts/users/peer.
The MIB data are needed to define and interpret uplink commands and downlink housekeeping. The housekeeping is interpreted using the binstruct module and are associated with a MIB via the TmVersion tables, which map a MIB to a period in time. We have different tm-version tables for each test environment (e.g., "ilt-qm", "ilt-fm", "ilt-par", "ist-fm"). You can set the tmversion with the following binstruct property:
hcss.binstruct.mib.pal.tm_version_map = ilt-fm
There is a dedicated tmversion table for the TB/TV test environment, it is required because the TB/TV data overlaps (in time) with the SOVT1 data but the applied MIB data differ. It contains references for the hifi_mib_2140 (before 4 Nov, 2008) and the hifi_mib_2143 (after 4 Nov, 2008). The MIB 2143 will also be applied for SPG processing during SOVT2, and contains downlink related updates. It can be downloaded from http://www.sron.rug.nl/hifi_icc/protected/userinfo/download/hifi-cal-lstore-19-Dec.tgz. If you are running Windows try the http://www.sron.rug.nl/hifi_icc/protected/userinfo/download/hifi-cal-lstore-19-Dec.tar version to avoid zipping issues.
This lstore contains the TB/TV tm-version table, which you can activate by setting:
hcss.binstruct.mib.pal.tm_version_map = ist-tbtv
in addition to your regular binstruct setup.
For SOVT1 science data the ist-fm tmVersion table is still used. If you are interested in the
satellite HK for SOVT1 data, you need to use the merged MIB available in the above hifi-cal pool
with the tmVersion table ist-all. In the near future, the merged MIB for analysis
of SOVT2 satellite HK wil be added to the "ist-all" tm version table. Applying merged MIBS for your
analysis requires the following binstruct property be set:
hcss.binstruct.services=herschel.binstruct.hpsdb.HpsdbPalServices.
The automatic use of the different tmVersion tables to associate HouseKeeping data via its mission configuration name will be available when SPR 2111 is implemented.
Running into the java heap space error can be a common affliction
when using HCSS software, especially when running the pipeline. Here are some things
to help:
Increase the memory available to JIDE by calling it as
jide -Xms512m -Xmx768m, the same can be done with HIPE.
However, this does not work for the user release and instead a line can be added
after the jhinit section in the JIDE script: options="-Xms512m -Xmx768m".
The "garbage collection" command System.gc() is also useful for clearing memory.
Swap Store Properties: It is possible to use the hard disk as swap space to preserve the memory available in Jide/Hipe. The following properties are defined to preserve computer memory. This becomes especially useful when pipeline processing long observations on a laptop, or on a pc with a 32 bit Operative System (TBC) and with average or limited memories capacities. However, any Task that uses or changes any HifiProduct (e.g., HifiTimelineProduct) will benefit from the use of swap space.
The following properties can be defined (in the myconfig file or using the
"Hifi Product" tab in propgen) to set or to configure the Swap mechanism.
hcss.hifi.pipeline.product.memory = true: Setting the value of this property to "true" enables the swap mechanism. Note that the default value is "false".
hcss.hifi.pipeline.product.swapstore = "swapStore".
This is the name of the LocalStore where the temporary data will be saved. The default location is:${user.home}/.hcss/lstore/swapStore .
hcss.hifi.pipeline.product.swapratio = 0.25: This property determines how much the swap mechanism is used and is used to set the threshold level of free memory. When a new dataset is set or retrieved from the HifiProduct, the HifiProduct will check the size of the dataset and the free memory in the system. If the condition:
is verified, then all the floating datasets contained in the HifiProduct will be saved in the swap store.
This property should have value between 0 and 1 and has a default value of 0.25.
If the value is 0 all datasets will be always stored in the swap store. This is safe, but it could create performance delay (in the time needed to process the pipeline) due to the access time to the hard disk.
In the case of long observations, setting the property to 1 could be dangerous because memory problems (like Java.heap.space exception), may still occur, although the pipeline will try to have the best performance possible.
hcss.hifi.pipeline.product.savedisk = true: This property determines whether an existing observation in the swap store should be overwritten or not. It is strongly suggested to keep the value = true, otherwise the space used in the hard disk will increase in proportion to the number of times a product is saved in the swap store.
SwapUtil Class: At the moment, the pipeline does not clean the swap store after the processing. To avoid the swap store completely filling the hard disk when many observations are processed, it is suggested one manually remove the swap store by either deleting the swapStore directory or in HIPE/JIDE:
from herschel.hifi.pipeline.product import SwapUtil SwapUtil.delete()
Table of Contents
HIFI spectra can be stored in various formats, and there are various ways to access them.
Level 0 spectra are obtained from the Versant database as follows:
before starting JIDE, set the relevant server and database in $HOME/.hcss/default.props or in $HOME/.hcss/myconfig. For example:
var.database.server = @iccdb.sron.rug.nl 0 READ
var.database.devel = ilt_fm_5_prop${var.database.server}
then in JIDE, set the obsid to be retrieved: obsid=268494774
get the housekeeping data: hk=AccessPacketTask()(obsid=obsid, apid=1026)
get the data frames: df=AccessDataFrameTask()(obsid=obsid, apid=1030).
The meaning of common APIDs:
apid=1028 HRS prime/horizontal polarization
apid=1029 HRS prime/vertical polarization
apid=1030 WBS prime/horizontal polarization
apid=1031 WBS prime/vertical polarization
next step is often the creation of a HifiTimeline product from the retrieved data frames and housekeeping data:
htp=HifiTimelineProduct(df,hk)
note that steps c., d., and e. can be very slow, if run on computers located far away from the Versant database, even for obsids with short integration times.
By default, spectral products are stored on a local disk in 'local store' format in the directory .hcss/lstore/. The user can modify the format and storage location by adding the following properties to the configuration files in the user's .hcss directory (usually .hcss/myconfig) before starting the JIDE session:
#store data in a local store ('lstore')
hcss.ia.pal.defaulttype = lstore
#default location of data storage
hcss.ia.pal.pool.lstore.dir = ${user.home}/.hcss/lstore
Regardless of data format or location, spectra are loaded into the JIDE session with the following three steps:
register the pools of interest, so they can be accessed. For example if pools 'test1' and 'test2' exist, they are registered as follows:
storage = ProductStorage()
pool1 = PoolManager.getPool("test1")
pool2 = PoolManager.getPool("test2")
storage.register(pool1)
storage.register(pool2)
obtain the unique 'urn' reference numbers of the spectrum products of interest. This can be obtained
interactively:
result=browseProduct(storage)
from the command line
query = Query("creator == 'my name'")
result = storage.select(query)
finally, load the spectral products. In this case, the first one in the list of returned urn's:
prod = storage.load(result[0].urn).product
HIFI spectra can be stored in various ways. While users may import data from the Versant database (section "Accessing Versant Database"), they usually store their products on a local disk in one of the following ways.
Spectra can be stored in 'local store' and 'simple pool' formats. The choise between those formats is for most users irrelevant ('local store' is in a fits format, which may be of advantage to some). The choice is made in the configuration files in the user's .hcss directory:
#store data as simple pool ('simple') or local store ('lstore')
hcss.ia.pal.defaulttype = simple
#default locations of data storage
hcss.ia.pal.pool.lstore.dir = ${user.home}/.hcss/lstore
hcss.ia.pal.pool.simple.dir = ${user.home}/.hcss
Regardless of format, spectra are exported in the same way in JIDE. First, create a pool called 'test', and then save the product in it:
storage = ProductStorage()
pool = PoolManager.getPool("test")
storage.register(pool)
storage.save(product)
Table of Contents
The HIFI data processing pipeline is used for processing data received from one or more of the four HIFI spectrometers on-board Herschel into a final product that is suitable for interactive analysis.
The HIFI pipeline can be run within the Standard Product Generator (SPG) of the Herschel Science Centre (HSC) or from the Herschel Data Processing interface (HIPE or JIDE) in a user's interactive session. It is designed to: obtain raw data from a database (or local store of data); remove instrument-related properties of the data; calibrate the resulting spectra; and, then combine the separate spectra from a single observation. The final product is dependent upon the observation mode and is either a calibrated spectrum, a set of co-added spectra or spectral 3D cubes. For more information about the pipeline steps and their results, please read the HIFI Standard Product Specification Document, or the HIFI Pipeline Specification document.
Calibration information is required in order to process data through the pipeline. You can access this
information through the Versant databases at the HIFI ICC or install the hifi-cal
database locally on your machine. See Chapter 1, Before Beginning for details.
We can run the HIFI pipeline within HIPE in the following fashion.
Select the "hifipipeline task" by double-clicking on it under the Hifi Category in the Tasks view. If you already have data for processing loaded into the session, then the "hifiPipeline task" appears in the "Applicable Tasks" folder, and you can open the Task dialogue from there. A "Hifi Pipeline" View is also available from the HIPE Window menu (under Show View) but it is not fully implemented yet.
Figure 3.1. Starting the HIFI pipeline task

Select "hifipipeline" from the Windows menus or the Task view. Once a HIFI observation context has been selected in the Variables view then the "hifiPipeline task" appears in the "Applicable Tasks" folder.
The default view allows you to do the following.
Re-process an already existing observation context, e.g. from the Herschel Science Archive, through the pipeline. You do this by dragging the name of the data to be reprocessed from the Variables view to the observation context bullet in the dialog that appears in the Editor view (see Figure below).
Select the spectrometers you wish to process data for by checking the desired instrument(s) and polarisation(s). Both H and V polarisations of both the Wide Band Spectrometer (WBS) and High Resolution Spectrometer (HRS) are checked by default.
Process data from the ICC database by typing in an obsid and database name. (Note that you must have permission to access the ICC databases and only read access is possible.)
Select which level to (re)process to (0, 0.5, 1, or all levels). The default set-up will pipeline data through all levels.
Figure 3.2. HIFI pipeline task: default view

In this example, an already processed PointModeLodeChop observation is being reprocessed up to Level 0.5, both polarisations of both spectrometers are included.
By clicking on the "expert" button, you may additionally control more detailed aspects of the pipeline set-up. The following items are available to experts only and are generally expected to be used only by HIFI calibration scientists.
If you wish to use your hifi-cal lstore, set this up as usual in myconfig and leave the "hifi cal" box blank.
Write out comments on the quality of data and processing steps by selecting "true" from the drop-down menu of the "quality" box.
To set the mode of the observation, type it into the "obsMode" box (required for ILT data).
To use a self-defined palStore, drag its name in from the Variable view.
Define the test environment by typing in the tmVersion (e.g., ilt-fm)
The execMode must be set to "INTERACTIVE" in order that the resulting ObservationContext be stored in your palStore.
Select "true" in the drop-down menu to clear the cache store.
Read in from file your own version of pipeline algorithms.
Toggle back to the default view by clicking on the "basic" button.
Once you have everything set, choose the name of the observation context that will be produced (or use the HIPE default), and click on "accept" to run the pipeline. The status ("running" if all is well, error messages if not) and the progress of the pipeline are given in the "info" section at the bottom of the Task dialogue.
The hrs and wbsPipeline tasks can be run individually to process data to level 0.5. The genericPipeline task can then be run to generate up to a level 2 product.
These pipeline tasks are run and set-up in the GUI in much the same way as the hifiPipeline task. Note, however, that an obsid and database cannot be passed to the genericPipeline task. Instead an HTP or ObservationContext resulting from one of the backend pipelines should be passed to it by dragging it from the variable list.
Figure 3.4. Running the wbsPipeline task
![]() |
The ObservationContext, "obs", generated can be passed to the genericPipeline task
Alternatively, you can (re)process a level 0.5 (1) product up to a level 1 (2) product using the level1Pipeline (level2Pipeline) task. The GUI interface is exactly the same for these pipeline tasks as for the genericPipeline task.
Note that you can also (re)process
an observation context up to levels 0, 0.5, 1 and 2 using the uptoLevel... buttons in the hifiPipeline
task (see Figure 3.2).
The individual steps of the HRS, WBS and Generic Pipelines are also found in the task pane when the HIFI Category is selected. By double-clicking on tasks you may use them to run through the pipeline step by step, in the order of you choosing. As an example, Figure 3.5 shows the GUI for the step in the WBS pipeline that subtracts the darks from the fluxes (DoWbsDark); three methods of dark subtraction can be selected from the drop-down menu. The HIFI Pipeline Specification document should be consulted for descriptions of each step of the pipeline, as well as the methods used and inputs required.
Figure 3.5. Running the pipeline step-by-step

It is possible to modify the steps taken at stages of the spectrometer pipelines.
Additionally, the wbsPipeline task has an option to be run step-by-step by selecting true from the step drop-down menu, (see Figure 3.2) .
An easy way to run the HIFI pipeline is to use the hifiPipeline task. The hifiPipeline task calls the hrs/wbs pipeline task, to process data from Level 0 to Level 0.5, followed by the generic pipeline task, which processes data to Level 1 (and beyond). The backend (WBS or HRS) pipeline tasks produce a HifiTimelineProduct, called level 0_5. The generic pipeline results in an observation context, which contains all the products generated by the pipeline task and is stored in a "hifi-pipeline" lstore (~/.hcss/lstore/hifi-pipeline).
Below are several examples showing how to use the HIFI pipeline task.
The hifiPipeline task is called into JIDE by:
from herschel.hifi.pipeline.PipelineTask import *
Run the pipeline and generate an observation context ('obs') from scratch (Figure 3.6):
obs = pipelineTask(obsid=268435583, db="ds1@iccdb.sron.rug.nl 0 READ")
If you do not specify the database with db=”...” then the default (var.database.devel) set in propgen will be used.
You can use a GUI to show a progress bar (test only with no clear information, at the moment).
obs = pipelineTask(obs=None, obsid=268435583, db="ds1@iccdb.sron.rug.nl 0 READ", gui=1)
To re-use the pipeline task, ensure that all the IO parameters are reset by setting obs=None
For ILT data, provide the obsMode name - the data itself does not have it:
obs = pipelineTask(obs=None, obsid=268516902, db="ilt_fm_5_prop@iccdb1.sron.rug.nl 0 READ", obsMode="HifiPointModeLoadChop")
Redefine the tmVersion if the selected database requires a different mission phase than the default in your binstruct property hcss.binstruct.mib.pal.tm_version_map (hifi default = "ilt-fm"):
obs = pipelineTask(obs=None, obsid=268439922, db="ilt_par_5_prop@iccdb.sron.rug.nl 0 READ", tmVersion="ilt-par")
The pipeline task automatically processes data from all four spectrometers. You can select an apid for processing to Level 1 (for now, processing to Level 0 always includes all available apids):
obs = pipelineTask(obs=None, obsid=268516902, apids=["1030"], db="ilt_fm_5_prop@iccdb1.sron.rug.nl 0 READ", obsMode="HifiPointModeLoadChop")
The pipeline can be re-run in various ways:
a. Re-run the pipeline, assuming Level 0 is available:
pipelineTask(obs=obs)b. Re-run for Level 0 too. (Note that all calibration and other products are not replaced):
pipelineTask(obs=obs, reprocessAllLevels=1)c. Or just re-generate Level 0:
obs = pipelineTask(obsid=268516902, obsMode="HifiPointModeLoadChop", uptoLevel0=1)d. You can then use this to process from Level 0 up to Level 0.5:
obs = pipelineTask(obs=obs, uptoLevel0_5=1)You can edit the algorithm of the pipeline tasks:
obs = pipelineTask(obs=None, obsid=268435583, db="ds1@iccdb.sron.rug.nl 0 READ", wbsAlgo=myWbsAlgo, hrsAlgo=myHrsAlgo,genericAlgo=myGenericAlgo)
The algorithms can be found in:
| WBS: {build_root}/lib/herschel/hifi/pipeline/wbs/WbsPipelineAlgo.py |
| HRS: {build_root}/lib/herschel/hifi/pipeline/hrs/HrsPipelineAlgo.py |
| Generic: {build_root}/lib/herschel/hifi/pipeline/generic/GenericPipelineAlgo.py |
And provide your own palStore to which the pipeline will write to:
obs = pipelineTask(obs=None, obsid=268435583, db="ds1@iccdb.sron.rug.nl 0 READ",
palStore = myStore)
Clear CachedStoreHandler to avoid a block due to none closed stores. Note, this closes ALL stores available in this cache and may affect other applications running in the session.
obs = pipelineTask(obs=None, obsid=268435583, db="ds1@iccdb.sron.rug.nl 0 READ", uptoLevel0=1, clearCachedStoreHandler=1)
Finally, if pipeline task is not behaving as you expect you could try a reset:
pipelineTask = PipelineTask()
The calibrated spectrum can be extracted from the resulting Observation Context as described in the section called “Accessing Spectra in Local Pool” and then viewed with several tools (see Chapter 4, Viewing Spectra).
In addition to using the HIFI pipeline task, one can run the underlying pipeline tasks to, for example:
Pass an observation context (from running the HIFI pipeline task) to the WBS pipeline task to re-run up to Level 0.5 for only one apid
from herschel.hifi.pipeline.wbs.WbsPipelineTask import *
newobs=wbsPipelineTask(obs=obs, apid=1030)
and then pass that to the Generic pipeline task
from herschel.hifi.pipeline.generic.GenericPipelineTask import *
newobs2=genericPipelineTask(obs=newobs, apid=1030)
Process a HifiTimelineProduct using your own HrsPipelineAlgo
from herschel.hifi.pipeline.hrs.HrsPipelineTask import *
newhtp=hrsPipelineTask(htp=htp, algo=myHrsAlgo)
Load dataframes and housekeeping (create an HifiTimelineProduct)
htp = wbsPipelineTask(obsid=268516902, db="ilt_fm_5_prop@iccdb1.sron.rug.nl 0 READ"))
As these examples illustrate, both ObservationContexts ("obs") and HifiTimelineProducts ("htp") can be passed to these tasks. The parameters used in the Section above can also be applied to these tasks.
Another way to run the pipeline is step-by-step using the three pipeline branches separately. This is the simplest way to view or modify the pipeline steps, which are contained in the following scripts:
The WBS pipeline is found in |
The HRS pipeline is found in |
The Generic (or AOT) pipeline is found in |
As an example, take (again) the simulated WBS-H (apid=1030) HifiPointModeDBS observation with obsid=268435583 from the simulator data database (ds1).
1. Load the WBS pipelinescript into your JIDE editor. The obsid, apid and database must be entered manually into the script, see Figure 3.7.
2. Step through the WbsPipeline.py script until the end, or play the entire script with the run-all button.
The output is a HifiTimeline product, which is stored in a simple Pool called simple.wbspipeline,
it and all the intermediate calibration products can be viewed in the DatasetInspector,
see Figure 3.8.
3. To run the Generic pipeline, load GenericPipeline.py into your JIDE editor and step through the script.
The Generic pipeline requires that data have an AOT-like structure.
Older obsids, such as gas cell data, do not have this structure and result in the error
herschel.ia.task.SignatureException:
params: Null is not allowed
The GenericPipeline.py script does not (yet) store the results in a Pool.
The HTP and other calibration products generated by the Generic pipeline can be viewed in the DatasetInspector, along with the calibration products from the WBS pipeline, see Figure 3.9.
Figure 3.9. Generic Pipeline Products
![]() |
Calibration products can be viewed in the Dataset Inspector when the Generic pipeline is run step-wise.
4. The Generic pipeline script cannot be played with the run-all button but can be run from the command line as follows:
from herschel.hifi.pipeline.generic.GenericPipelineAlgo import runGenericPipeline
newhtp=runGenericPipeline(htp, None)
However when the Generic pipeline is run this way, no calibration products are generated and only the resulting HTP can be viewed in the DatasetInspector, see Figure 3.10
Figure 3.10. Running the Generic Pipeline from the command line
![]() |
The calibrated spectrum can be viewed in the Spectrum Explorer.
The spectrum in Figure 3.10 is viewed in the Spectrum Explorer within the DatasetInspector. See the section called “The SpectrumExplorer Package” in this document for information about using the Spectrum Explorer, and for information about the DatasectInspector see Chapter 4.14 (The DatasetInspector) in A Basic User's Manual Herschel Data Processing, which can also be found in the documentation within JIDE.
Table of Contents
HIFI spectra can be visualised in several ways, at various levels of sophistication and user-friendliness. Here the PlotXY and SpectrumExplorer packages are described.
PlotXY() is the basic package to plot arrays of data points in the HCSS, and it can be used to plot HIFI spectra as well. It has a lot of options, making the plots highly configurable. Here is an example of plotting a HIFI spectrum:
Get the frequency and flux data to be plotted from the spectrumdataset 'sd':
freq=sd.getWave().get(0)
flux=sd.getFlux().get(0)
The simplest possible plot:
out=PlotXY(freq, flux)
When plotting multiple spectrum datasets, say 'sd1' and 'sd2' in one figure:
#get the wavelengths and fluxes to be plotted
freq1=sd1.getWave().get(0)
flux1=sd1.getFlux().get(0)
freq2=sd2.getWave().get(0)
flux2=sd2.getFlux().get(0)
#create the plot variable
p=PlotXY()
#create the plots in batch mode
p.batch=1
#define the layer variable
ll=[]
#remove any non-numbers (NaN's, Infinites etc.)
valid=flux1.where(IS_FINITE)
#create layer for first plot
l=LayerXY(freq1[valid],flux1[valid])
#append to layer variable
ll.append(l)
#repeat the above for the 2nd plot to be overlaid
valid=flux2.where(IS_FINITE)
l=LayerXY(freq2[valid],flux2[valid])
ll.append(l)
#define the plot layers that have just been created
p.layers=ll
#get out of batch mode. This actually creates the plot
p.batch=0
And this is how some common features of the plot are modified.
p.setYrange([0, 1.5])
p.setTitleText("This is an example plot")
It is also possible to display spectra without taking apart the data format as is
described in the previous section. All Herschel spectra types can be displayed with the
SpectrumPlot package.
If spectrum is a Herschel Spectral type (Spectrum1d, Spectrum2d) then:
splot=SpectrumPlot(spectrum, useFrame=1)
will simply display the spectrum along with some standard header information.
The useFrame=1 allows for the possiblility of creating a plot without actually viewing it
at first, but as the last step. The SpectrumPlot module is build on PlotXY, and so many of the features
you would use in PlotXY you can also use for SpectrumPlot. Below are a few
examples:
from herschel.ia.toolbox.spectrum.gui import SpectrumPlot from herschel.ia.gui.plot.renderer.StyleEngine.ChartType import HISTOGRAM,LINECHART # # Creating the plot sp=SpectrumPlot(spectrum,useFrame=1) # # adding a second spectrum to the plot sp.add(spectrum2) # #Start fresh again p = SpectrumPlot(spectrum, useFrame=1) # #get graphs g0 = p.getGraphs()[0] g2 = p.getGraphs()[2] # #display as line graph or histogram g0.layer.style.chartType = HISTOGRAM g2.layer.style.chartType = LINECHART # #add annotations g0.layer.addAnnotation(Annotation(4000,1,"My annotation")) g0.layer.addAnnotation(Annotation(5000,0.98,"My annotation")) # #select a range of data g0.layer.xaxis.addMarker(AxisMarker(4200,4400)) g2.layer.xaxis.addMarker(AxisMarker(6000,6500))
These last lines will produce the following plot:
The SpectrumExplorer package is based on the PlotXY package, but allows the user to visualize SpectrumDatasets in a friendlier, interactive way. To use it, open the DatasetInspector and click on a SpectrumDataset in the left panel. This will plot the spectrum. If the SpectrumDataset is wrapped in a product, you will have to extract it first. Click on the product in the left panel of the DatasetInspector and then click on a SpectrumDataset in the right panel with the right mouse button. Then click on the blue pop-up window "add to left side". Alternatively, determine the index of the SpectrumDataset you want to extract and at the JIDE prompt enter:
myspectrumdataset=myproduct.get(index)
Then clicking on the SpectrumDataset will show something like this:
In the example, the different colors indicate different WBS sub-bands. Individual sub-bands or individual scans can be plotted by clicking on the appropriate boxes in the bottom panel and removed by double-clicking. Any plot parameter (plot range, titles, colors etc.) can be modified using the right mouse bottom in the same way as for the PlotXY() package. In addition the plot can be modified interactively after clicking the appropriate action button at the top panel. From left to right:
button 1: highlight a spectrum (or WBS sub-band) by moving the mouse over it and click the right mouse button to change its color, description, or remove it.
button 2: change the horizontal and vertical plot ranges by drawing a rectangular box using the left mouse button. Also, one can scroll the spectrum along the horizontal and vertical axes by clicking on an axis with the left mouse button and then moving the mouse or using the mouse wheel. The mouse wheel can also be used to (un)zoom the spectrum.
button 3: select specific data points for flagging or removal.
button 4: highlight spectral ranges by using the left mouse button. This will create a vertical grey bar. The range can be removed or inverted using the right mouse button. In the near future all sorts of tasks can be applied to the marked area(s), e.g. flagging datapoints and fitting lines and Gaussians. [note: button 4 will be removed in future releases and the ranges can be defined using the middle mouse button]
button 5: pan through the spectrum by clicking the left mouse button and moving the mouse.
button 6: click on a spectrum (or WBS sub-band) and drag it to another or a new panel (however, dragging to the left or top of the first panel is not possible).
button 7: only show the active plot panel, and change the axis ratio in order to fit the screen. Click button 7 again to show all panels.
button 8: re-set x- and y-axis plotting ranges of the active plot window to default values
Finally, the SpectrumExplorer package is still under development. Future developments include:
clicking on product will plot all SpectrumDatasets included (likely a HIPE functionality, not JIDE)
apply a filter to the meta data and only plot the applicable spectra (e.g. a certain chopper position)
applying functions to greyed-out spectral regions (see button 4 above)
saving modified SpectrumDatasets back into the session, e.g. with interactively masked points removed.
overplotting multiple SpectrumDatasets.
Table of Contents
Spectral features (baseline, lines and noise) are fitted using the spectrum fitting toolbox in the HCSS.
The data that is used by toolbox can be any Java or Jython object, as long as it implements the SpectralSegment interface (e.g., extracted from a Spectrum1d object). An example of a SpectralSegment could be the spectrum from one subband.
Select the spectrum to fit from the variable list and then double click on the task fitSpectrum (Figure 5.1).
Apply a model to the spectrum via the GUI that pops up.
All 1D models that are in ia.numeric.toolbox.fit can be selected from the drop-down box 'Use model'. The default is a Gaussian ("gauss") model, for which the 'Height of peak', 'X-Position of peak', and 'Width (sigma)' must be defined. The height and width have the value '1' already filled in, supply a value for the position and click RUN (Figure 5.2).
A Lorentzian ("lorentz") model can be fitted in a similar way. If you select a polynomial ("poly") fit, then only the order of the polynomial needs be defined. The 'fix' tickboxes can be used to fix the value of the parameters (Figure 5.3).
If any other model is selected, the GUI will look like Figure 5.4 . If the model requires constructor parameters (see the JavaDoc of the model), the 'Constr Parms' field must be filled with a comma-separated list, for example, the "power" model needs a degree. If no constructor parameters are needed, as for the "sinc" model, leave the field empty. In the 'Parameters' field the fit parameters must be filled in with a comma-separated list - be sure to give the correct amount of parameters (see the JavaDoc). In the 'Fixed' field the parameters that must be fixed can be listed in (guess what) a comma-separated list. If the first and third parameter must be fixed, fill in: 0,2.
This 'other model' GUI is not yet very user friendly - but GUIs, including 'help' buttons, will be implemented for each model.
A fit can be applied over a specified range.
Click on 'Add Window' to define the fit X-range (between 'from' and 'to'). Up to five ranges can be defined by clicking on 'Add Window' again (Figure 5.5).
The result of a fit is a 'fitResult' variable. Double clicking this variable opens a view window with the data and fitted model (top) and the residual (bottom). A table of the fitted parameters and their standard deviations can be seen by then clicking on the 'ShowFitResult' task (Figure 5.6). If the fitResult contains several models, the parameters for all models are listed here.
Figure 5.6. Tasks available after fitSpectrum
![]() |
Click on ShowFitResult to see fitted parameters and their standard deviations
Another model can be applied to the result of a fit. So you can, for example, fit the baseline and then fit a spectral feature.
Click on a 'fitResult' and then again on the 'fitSpectrum' task and follow the proceedure given above. This results in another fitResult variable to which you can apply another fit model, and so on.
Once satisfactory models for all spectral features have been found, all the models can be applied to the original data.
Click on your final fitResult variable and then click again on the fitSpectrum task. The GUI contains a checkbox 'global fit' (Figure 5.7), check this and click on 'RUN'. No new model can be added at this stage.
Download the toolbox into the session, note that in JIDE it is called SpectrumFitter rather than fitSpectrum!
from herschel.ia.toolbox.spectrum.fit import SpectrumFitter from herschel.ia.toolbox.spectrum.fit.testdata import MakeData
For demonstration purposes, we will use MakeData to create some test data to fit.
data=MakeData(7) data.addNoise(10) #instantiate the fitter sf=SpectrumFitter(data)
A plot window should look similar to that shown in Figure 5.8.
The SpectrumFitter is an interactive tool and is best used in conjunction with the SpectrumModel tool, which allows you to select (and change) models and fitting parameters. The three models you are most likely to use are Gaussian, Lorentzian and Polynomial; the model fits, their parameters, and their usage in the SpectrumFitter tool are summarized in Table 5.1:
Table 5.1. Model fits, their parameters and usage in the SpectrumFitter tool
| Model | Mathematical fit | Parameters | Usage |
|---|---|---|---|
| Gaussian | a0 = amplitude of line |
sf.addModel('gauss', [a0, x0, s0])
| |
| x0 = location of line peak | |||
| s0 = width of line (sigma) | |||
| Lorentzian | p0 = amplitude of line |
sf.addModel('lorentz', [p0, p1, p2])
| |
| p1 = location of line peak | |||
| p2 = half width at half maximum of line | |||
| Polynomial | f(x) = c0 + c1x + ... + cnxn | n = order of polynomial |
sf.addModel('poly', [n], [c0, c1, ..., cn])
|
| c0 .. cn = polynomial coefficients |
Note that you must know (roughly) where you expect a spectral feature in your data to be, in addition to its expected shape and approximate shape parameters. So, an initial guess is required - if this guess is completely wrong you may end-up fitting noise rather than your spectral lines.
Now, fit first the baseline with a polynomial and then fit the line with a Gaussian.
#First the baseline
# Apply the model
model=sf.addModel('poly', [2],[0,0,0])
# Do the fit
sf.doFit()
# Inspect the residual after the baseline is removed
sf.residual()
# Keep the fit
sf.fitOK()
#Now the line
sf.addModel('gauss', [1.0,30,0.1])
sf.doFit()
sf.residual()
sf.fitOK()
These steps result in the plot below. A black line (not seen here) displays the model and is replaced by a green line showing the fit (the Gaussian model here). The red line is the final fit for the entire spectrum. The residual is shown in a separate plot.
It is possible to do both fits at the same time, globally, since the instance of our SpectrumFitter remembers what it has done so far.
sf.doGlobalFit()
It is also possible to mask data. The following will do a polynomial fit only using data from 0 to 20 and from 40 to 100.
model=sf.addModel('poly', [2],[0,0,0])
#after you've created the model, now add the masks.
model.setMask(0,20)
model.setMask(40,100)
To best see how this works, include this masking in the example given above.
The fitted model parameters and their standard deviations are printed to screen with:
print sf
It is possible to manipulate the models produced by SpectrumFitter in various ways:
If you wish to change the initial parameters of any of the models (model = sf.addModel(...)),
use setParameters:
model.setParameters([...])
A new fit will be made on the fly.
There are two ways to remove models:
sf.removeModel(m)
Or:
m.remove()
Subtract the model from the dataset:
sf.subtractModel(m)
This also removes the model from the fitter tool.
Once you are satisfied with a fit, you can set the fitted parameters as the default for the models:
m.useResults()
This may be useful when using the same models for a following dataset.
To apply them to a different dataset:
sf.setData(otherData)
Note that this replaces the data held in the SpectrumFitter with the SpectralSegment held in the variable 'otherData'. Once again, the fit will be redone on the fly.
Table of Contents
The IA User's Manual contains the most updated information for the spectrum arithmetics module. Please see the IA User's Manual for more up-to-date information.
The spectrum arithmetic toolbox allows to combine Herschel
spectrum data. Operations are performed either on subclasses of spectrum
datasets (Spectrum1d, Spectrum2d), on cubes
(SimpleCube, SlicedCube), or on products containing such
data structures (e.g., HifiTimelineProduct).
Operations on Spectra include Selection and Arithmetic Operations.
Selection: Provide means of selecting those spectra that can be combined. For instance cold-load spectra, ON spectra, etc. Selection can be applied to datasets, such as rows of a Spectrum2d, or to tables within a product, such as datasets included in a HifiTimelineProduct.
Arithmetic Operations: Provide means of combining the selected spectra. This includes:
Basic arithmetic operations such as addition, subtraction, multiplication, or applications of scalar functions.
Statistical operations such as mean, median, variance, standard deviation or percentiles for samples / selections of spectra.
Data transformations such as smoothing or frequency re-sampling.
It is planned that the arithmetic toolbox will provide generic functionality for all instruments (HIFI, PACS and SPIRE). Instrument-specific behavior will be pre-configured by defaults in the system but can also be overwritten by the user.
We present the power of the toolbox with a few code examples. Assume
we have started a jide session and loaded a Spectrum2d
dataset with name 'data' from a local pool or a database.
We might want to work only with a sub-set of the spectra included in
our data. For a Spectrum2d
this means we have to (1) select specific rows from the data and (2)
combine them into a new dataset by applying some arithmetic
operations on the selection. Task (1) is performed with the
SelectSpectrum
task,
from herschel.ia.toolbox.spectrum import SelectSpectrum
>
The SelectSpectrum-task can be configured and used in many different ways. A frequent usage is to identify all the rows of the dataset that have a specific value in a particular column:
ds1 = SelectSpectrum()(ds=data, selection_lookup={"bbtype":[3260]})
The example above selects all the rows with a value=3260 in the column named 'bbtype'. Hence, the selection is performed by using the keyword selection_lookup in the call of the task, using what is called a python dictionary. This py-dictionary contains the name of the attribute to look up as key (column name) and the attribute value as value. All the rows in the resulting dataset ds1 have values 3514 in the bbtype column.
After selecting the data, we can move to task (2), the application of some arithmetic operations to the selected spectra. For example, if we now want to average the selection, we can invoke the AverageSpectrum task:
from herschel.ia.toolbox.spectrum import AverageSpectrum
avg21 = AverageSpectrum()(ds=ds2)
The selection explained in task (1) can also be included in the average spectrum task, thus allowing to perform selection and averaging in one step:
avg22 = AverageSpectrum()(ds=data, selection_lookup = {"bbtype":[3413],"buffer":[1]})
This result is identical to the separate operations. It includes a single row with the average flux. The resulting dataset contains exactly the same columns as the input dataset. Thus, what values should we fill in the columns not affected by the operation? This is determined by a default action that depends on the input data type (sub-class of Spectrum2d in our example). For the Spectrum2d, the default action consists of copying the values found in the input spectrum.
This way of processing the data is general: We always try to keep as much information as possible. All columns and also the meta data are set in a type specific, instrument specific, or user specific way. The output data type is the same as the input data type.
Other arithmetic operations are available such as pair operations (subtract, divide, pair-wise add/multiply) and scalar operations (add/subtract or multiply/divide by a scalar quantity). Here is an example that shows how to use the subtraction:
from herschel.ia.toolbox.spectrum import SubtractSpectrum
diff12 = SubtractSpectrum()(ds1=data1, ds2=data2)
Here, the datasets data1 and data2 either must have the same number of rows, or one of them must have only a single row. If they have the same number of rows, the subtraction is carried through for the flux data on a row-by-row basis. If the second contains only one row, this row is subtracted from all the rows in the first dataset (or the other way around).
The same task can also be used for subtracting a scalar:
ds_m2= SubtractSpectrum()(ds=data, param=2)
The use of the DivideSpectrum -task is identical:
from herschel.ia.toolbox.spectrum import DivideSpectrum
ratio12 = DivideSpectrum()(ds1=data1, ds2=data2)
ds_d2 = DivideSpectrum()(ds=data,param=2)
Similarly, the
from herschel.ia.toolbox.spectrum import MultiplySpectrum
from herschel.ia.toolbox.spectrum import AddSpectrum
can be used.
The configuration of the arithmetic operations can be customized. Here are a few examples:
Specifying operations on additional attributes (columns in a Spectrum2d)
Specifying how to combine flux, weight and flag data included in datasets.
Specifying, as already mentioned above, other 'selection models' suited for particular situations (e.g. column values in a given interval, column values in regions around given discrete values, column values above/below thresholds, etc.).
For each dataset type, a default behavior is defined for 1) and 2).
Operations are available both at the Task level and at the Java level. All operations follow the same basic idea as shown above. This is just a summary of the available tasks. The Java Classes are discussed in the Developer's Manual.
SelectSpectrum
AverageSpectrum
AddSpectrum
SubtractSpectrum
DivideSpectrum
MultiplySpectrum
Table of Contents
Many of the spectral viewing, arithmetics and fitting tasks discussed in previous sections can be called from within the ClassLikeTool environment. ClassLikeTool reads products from and writes them to a local pool.
Level 0, 1, or 2 products stored in a pool on a local disk can be accessed by ClassLikeTool by entering the following on the HIPE or JIDE command line (or running it from a jython script):
#import the java class from herschel.hifi.dp.gui.spectrum.classliketool import DoClassLikeTool #define the pool name that exists on your local disk obsid="268435841_sg" #register the pool storage = ProductStorage() pool = PoolManager.getPool(str(obsid)) storage.register(pool) #open ClassLikeTool on the pool doclassliketool=DoClassLikeTool(storage)
A window will appear on your screen, with buttons to select, plot, fit, and perform various arithmetics operations, which are discussed in the next sections.
Next to the plotting window, three tabs are visible ('Select', 'Arith.', and 'Fit'), which are each discussed in the next three sections. The area below the plotting window shows a number of buttons and boxes that can be used when any of the tabs on the right are selected:
The function of the buttons and boxes shown above is as follows:
Change the units of the bottom X-axis (top X-axis not yet implemented), selecting the desired unit via the pool-down menu.
Clip data or spectra (see next section).
Change the line chart via the pull-down menu (line or histogram).
Change the width of the plotted lines (use with caution, very thick lines may decrease performance).
The left mouse button can be used to zoom in on part of the spectrum. Click the "Zoom" button to un-zoom, or the double left arrow button (resp. double right arrows) to go back to the previous (resp. next) zooming panel.
Click the "Save Pool" button to save the data as they are seen on the screen to a new pool that can be open by ClassLikeTool.
Click the "Generate Script" button to save a script of all performed commands [not yet implemented].
Mark one or any number of areas on the plot window with the middle mouse button. The toggle button "Show/Hide" allow to show or hide the selected area(s) and the "Clear" button clear all selected areas. Double left clicking on any greyed area makes it disappearing.
Any number of points or a whole spectrum can be clipped by selecting the "Spectra" or "Data" button at the "Clip" section in the bottom-left panel. The user then selects the data/spectra to be removed drawing a rectangle over the data while pressing the left mouse button. Data points are automatically removed, while selected spectra first appear in black on the plotting window, the user can accept the selection (and therefore the clipping) pressing the "Remove" button. The "Undo" button allows to undo the clipping both for data and spectra.
The first tab on the right panel of the ClassLikeTool window allows one to select datasets in the pool. As shown in the HIPE screen-shot below a selection can be made between spectrometer backend, obsid, WBS sub-band ("1-4" selects all sub-bands, "1,4" sub-bands 1 and 4), and spectrum dataset type and integration number (in this case numbers 0-10 out of 72 integrations). Clicking the large "Display" button at the bottom creates the plot in the left panel.
In the example shown, the x-axis units are selected with the "X Bottom" button: IF, USB, LSB frequencies or a velocity scale with respect to a frequency entered by the user in a pop-up window. The plot window makes use of the PlotXY package, and thus the plot can be modified using the options under the right mouse button.
A numbers of operations can be performed on the spectra that have been selected in ClassLikeTool (see previous section), by clicking on the "Arith." tab at the top right (see figure below). A particular operation can be un-done with the "Undo" button at the bottom right. If multiple operations have been performed, only the last one can be un-done. The original spectra can still be retrieved using the "Original" button. Plots of the spectra before and after the operation can be made with the "Overlay Previous" button. The following operations are possible using the buttons from top to bottom:
add or subtract a scalar. Manipulation of vectors, i.e. adding or subtracting different numbers at each channel, has not been implemented yet.
multiply or divide by a scalar. Also here manipulation with vectors, has not been implemented yet.
resample to a grid with a user-defined spacing, using flux-conservative interpolation.
average the selected spectra either with equal weights for all spectra or with weights derived from the system temperature (once these are implemented in the pipeline) and user-defined flags. If the spectra are not sampled on the same grid, the user will be warned that a re-sampling will have to be done first. Three averaging options are available:
all: average every selected dataset, incl. HRS and WBS. The output will be 1 spectrum.
datasets: average datasets per backend. The output can be up to 4 spectra: HRS H, HRS V, WBS H, and WBS V.
WBS/HRS: average all datasets per spectrometer type. The output will be up to 2 spectra, one for HRS and one for WBS
smooth the spectra either by convolution with a box or a Gaussian of which the user can specify the width in MHz.
stitch WBS or HRS sub-bands. This can only be done if the selected spectra have been averaged (e.g. using option 4 above), or only one spectrum was selected.
take the FFT of the spectrum.
fold frequency-switched spectra [not implemented yet]
integrate over spectral regions selected using the "Axis Marker".
determine the rms over spectral regions selected using the "Axis Marker".
divide the spectrum over the main beam effiency (b_eff).
The following HIPE screen-shot shows buttons of all arithmetics options. The integral and RMS noise values were calculated for the areas indicated with vertical grey bars. The latter were defined using the middle mouse button and displayed using the 'Axis Marker' box.
With the third tab on the right, the spectrum can be fitted with polynomials, Gaussians, and Lorentzians. Other functions, including sine waves and Voigt profiles will be added in the future. Fits can be made to regions selected with the middle mouse button and viewed using the 'Axis Marker'. The user can opt for the Amoeba and Levenberg fitter methods, that sometimes give slightly different results. The particular functions to be fitted are selected using the 'Add Component' button. In the example shown below, two Gaussians were fitted. For this instructive purpose, the peak positions ('x0') were assumed to be known to be 0.0 and 200 km/s, and they were fixed to those values by clicking the boxes now showing the 'v' signs (note: presently the option of fixing parameters only works with the Amoeba fitter). Approximate values for 'I0' and 'FWHM' were automatically assigned with respect to the grey area selected with the mouse. The 'Fit' button at the bottom was clicked several times in order to iterate to the best solution.
In the HIPE screen-shot below, the thick blue line represents the data and the green line the Gaussian fits. The residual of data minus fit can be plotted (not done here for clarity) using the "Overlay residual" button.
The Housekeeping (HK) section walks through some basic guidelines to accessing, viewing, and exporting HK data. HK packets have some similarities in structure with the dataframes packets.
Most of the examples are just few lines of the DP code that can be entered at the JIDE prompt, or code fragments or routines designed to serve as an examples for your own programs.
Some limited examples for using the different tasks are given in the help pages for all tasks in Chapter 3. "DP Commands" of the User's Reference Manual.
Please note that it is desirable that trend analysis is carried out at the WBS/HRS pipeline level, as we want to follow trends at the instrument level.
HK data can be both digital and analog.
There are two main methods for obtaining dataframes: by command line access or by a DataSelector GUI. In the first case you will get the frames and put them into an array or table, in the second a more sophisticated method will allow you to access the database (test only with no clear information, at the moment). Below, a basic description for accessing and retrieving dataframes is given.
In 1.1.1 already a brief description of how to get level 0 data is given. Data frames in the database are basically accessed after setting the relevant server and database. Afterwards:
In JIDE, set the obsid to be retrieved: obsid=268494774
Get the HK data: hk=AccessPacketTask()(obsid=obsid, apid=1026)
Get the data frames in the database: df=AccessDataFrameTask()(obsid=obsid, apid=1030)
However, with the task ObservationSelector the data frames can be obtained in two additional different ways,
by choosing them from the database viewer and selecting the obsid:
dataframes = ObservationSelector() (obsid, spectrometer, gui).
To display the values of all digital and analog HK values of the HRS into a JTable:
from herschel.hifi.hrs.task.import *
hkDataset = HrsGetHKTask() (apid=1028, obsid=268494774)
HrsViewHKTask() (model="FM", apid=1028, hk=hkDataset)To print to an ASCII table file information on periodic HK for LO during an observation:
lo = LO_HK_dump()
t(268494774,5,10)
To print to ASCII table file spectra taken with either WBS. ASCII table spectra also accompanied by a set of useful HK data
t = WBS_Export_ascii()
t(observation=268494774, plot="V")
To read and select data set for an spectrometer (wbs or hrs): observation=ReadObservation()(obsid=5861, spectrometer="wbs-h")
To retrieve system temperature and bandpass/gain determined in a Hot-Cold measurement:
allhot=SelecScans()(spectra,validbbs=Intlb([2223])
allcold=SelecScans()(spectra,validbbs=Intlb([2222])
hot=SingleHifiSpectrum(allhot)
cold=SingleHifiSpectrum(allcold)
ct=ComputeTsys()
noise=ct(hot,cold,eta_hot=0.96)
gamma=ct.gamma
To compute the system temperature using Hot-Cold load results from the HRS:
t = RadioMetry()
t(observation=268494774)
To compute system temperature using Hot-Cold load results from the WBS (across the IF band):
t = RWbs_HotCold_fast_bare()
t(observation=268494774)
To compute the WBS IF noise temperature (across the IF band):
t = Wbs_IFNoise()
ifout=t(observation=268494774)
From the retrieved data frames and HK data a HIFI timeline product can be created. A HifiTimelineProduct is an extension of HifiProduct. It also contains a summary table and a listing of the types of the HSDs.
df = AccessDataFrameTask()(apids=[1030,1031],obsid=268435473)
htp=HifiTimelineProduct(df,hk)
To get start and end time: FineTime start [INPUT, OPTIONAL, default=null] and FineTime end [INPUT, OPTIONAL, default=null], respectively
To retrieve the HK packets containing Gascell information from a database:
Use AccessPacketTask on apid 2025 for the provided obsid. You should also provide a database name (full version, including the server name). If you do not, then the query will be applied to the current database. This task instantiates AccessPacketTask only once.
Example:
obsids = [268449660, 268449664, 268449665, 268449670] database = 'ilt_fm_2_prop@iccdb.sron.rug.nl 0 READ' accessGascellHk = AccessGascellHk() accessGascellHk.db = database# for obsid in obsids:hk_pack = accessGascellHk(obsid = obsid) print 'obsid', obsid, 'has', len(hk_pack), ' gascell packets.'
To convert HK packets containing Gascell information into a table use the hk_tools.spy and tables_tools.spy routines to create the table. The list of the gascell HK parameters which are tabulated are given in the Chapter 3 of DP Commands.
To generate a table data set of the HK data: TableDataset selected [OUTPUT, OPTIONAL, default=None]
To create a tabla data set of HRS HK converted values (with as many columns as HRS HK mnemonics exist):
from herschel.hifi.hrs.task import *
hk = HrsGetHKTask()(apid=1028, obsid=268435622, type="All")
Helper class for defining the GUI components for AccessHkParamTask:
ahpt = AccessHkParamTask()
ahpt(db=ilt_fm_5_prop@iccdb.sron.rug.nl, obsid=268505167, apid=2017, params = ["FPU_room_temp", "FPU_shutter_temp"])
To store HK information in a PAL: Hk_Store()(obsid=1732, lstore="HK_data", desc="Diplexer scan HK")
As mentioned in the Chapter 4 of the Generic Pipeline description, some HK data are important for the Generic Pipeline. The following HK data are important when processing the generic pipeline:
LO-frequency ("LoFrequency"): Generally, for grouping comparable spectra or to check and identify phases in the FSwitch observations.
Chopper position ("Chopper"): To check and identify phases.
Buffer ("buffer"): Alternative to check and identify phases.
Observation time ("obs time"): For interpolating possibly drifting intensity scales (hot/cold measurements) or "background" obtained by blank sky measurements.
Hot/Cold load temperature("hot_cold"): Used in the intensity calibration (determination of the bandpass and the system temperature).
A list of helpers class for defining the GUI components for the following generic tasks:
AccessDataFrameTask: Allows access to DataFrames in the database. Helpers: AccessComponents, AccessDataFrameComponents.
AccessPacketTask: Allows access to TmSourcePackets in the database. Helper: AccessComponents, AccessPacketComponents.
AccessHkParamTask: Allows direct parameter retrieval from TmSourcePackets in the database(s). Helper: AccessHkParamComponent.
A description of the helpers is given in Chapter 3 of the User Manual.
Some parameter names:
FPU_room_temp
FPU_shooter_temp
Example of how to retrieve them:
from herschel.hifi.generic.task import AccessHkParamTask
ahpt = AccessHkParamTask()
ahpt(db=ilt_fm_5_prop@iccdb.sron.rug.nl, obsid=268505167, apid=2017, params = ["FPU_room_temp","FPU_shooter_temp"])
composite=ahpt.paramresult
print composite
1.- Accessing HK information for a given ObsID. Product: Table [time, FPU Temperature]
HcssConnection.get() task passes all the dataframes into the user's JIDE session. This can be memory intensive.
# Import needed packagesfrom herschel.access import * from herschel.access.util import * from herschel.binstruct import * from herschel.pus import *# Access HK packets associated with ObsID = 1400pk = PacketAccess(1400)# Connect to the default database to find the packetshk_set = HcssConnection.get(pk)# Create an empty Java array list - needed for the PacketSequence routine below.arrList = java.util.ArrayList()# Loop around adding the HK dataset into the array arrListfor x in range(len(hk_set)): arrList.add(hk_set[x])# Look at our arrayprint arrList# ...but to get something sensible we need packets in a time order.pseq = PacketSequence(arrList)# Get a listing of the parameter types contained inprint pseq# Find packets in the sequence which contain information on temperatures within the focal plane unit (FPU)seq_FPU_Temp = pseq.select(TypeEquals("FPU_Temperatures"))# Find out parameters contained in the selected packets by obtaining the HK parameter names from the first selected packet in the sequencepar_FPU_Temp = seq_FPU_Temp[0].getParametersContained()# Print out to the DP session the names of all the parameters containedprint par_FPU_Temp# Choose the FPU Temperature parameter you want to get info on...and get a time ordered set of HK data for it # The output file plot_fpu_hk is a TableDataset with one column for time (a Finetime of microseconds since 1 January 1958) # and one for the value of the parameter (RAW rather than engineering value). Here we choose the parameter FPU_b_body_top for the table # output and get the converted values (in degrees K)plot_fpu_hk = seq_FPU_Temp.getConvertedMeasures(["FPU_b_body_top"]) time = Double1d(plot_fpu_hk[0].data/1000000.0) # puts time into seconds data = Double1d(plot_fpu_hk[1].data)# Plot the timeline of the HK data over the time period of the observation (obsid=1400) by plotting the tablep = PlotXY(time, data, style=Style(line=8, color=Color.black))# Resize the windowp.height = 400 p.width=600# Give a layer/legend name...p[0].name="time plot"# ...and add a titlep.title.text="FPU temperature"
2.- Accessing HK data for a given time period (covering several ObsIDs). Product: a plot time vs. Mixer Voltage.
If you choose to sample long time period data this will prove to be memory intensive.
# Import needed packages for handling databases and HK datafrom herschel.access import * from herschel.access.util import * from herschel.binstruct import * from herschel.pus import *# And this package deals with times.from herschel.share.fltdyn.time import *# Enter a start and stop time for HK information. We enter Java Dates, given as year (-1900), Month (-1), # day, hour, minute, second. Our start_time is therefore 01:10:00 on 25 October 2004start_time = java.util.Date(104, 9, 25, 1, 10, 0)# stop_time is 01:15:00 on the same daystop_time = java.util.Date(104, 9, 25, 1, 15,0)# Need to convert final numbers into a FineTime used in database.start_1 = DateConverter.dateToFineTime(start_time)# Date/time of start for plotted dataprod_date = DateConverter.fineTimeToDate(start_1)# Ditto for stop timestop_1 = DateConverter.dateToFineTime(stop_time) end_date = DateConverter.fineTimeToDate(stop_1)# Initialize some parameterspk=0 hk_set = 0# Get object ready for sorting packets.pseq = PacketSequence()# Set up the query for accessing packets of HK data. Here we ask for packets with an APID of 1026, which carries # HIFI HK data. The database identified by the user's properties is accessed for packets of this type # between the given start and stop FineTimes.pk = PacketAccess(1026,start_1,stop_1)# Now we know where to look, we can get the packets! First we create an array with the packets inhk_set = HcssConnection.get(pk)# ...then we loop over the array to get the contents and # put packets into our packet sequencefor x in range(len(hk_set)): pseq.add(PusTmSourcePacket(hk_set[x].getContents()))# Now we get the parameters in the packets that we can plot.seq_HIFI_HK = pseq.select(TypeEquals("HIFI_HK_rev_3"))# Pick out some of themmnemonics = ["HF_AH1_MXMG_V", "HF_AV1_MXMG_V"]# ...and get their converted (physical unit) measurements. "plot_HIFI_HK" is a TableDataset with a first column measuring time # and the next 2 columns holding the HK parameter values at those times. We can now plot any of the parameters versus # time, or against each other, by picking out the appropriate column of the table.Plot_HIFI_HK = seq_HIFI_HK.getConvertedMeasures(mnemonics)# This is what to do to set up the plot. Since time is in microseconds we convert it to seconds first. # Get the first column and divide by 1 milliontime = plot_HIFI_HK[0].data/1000000.0# Measure time on the plot from the beginning of the observation....We subtract the initial time valueplot_time = time - time[0]# Plot the two voltages contained in columns 2 and 4h_voltage = plot_HIFI_HK[1].data v_voltage = plot_HIFI_HK[2].data# Plot the datap = PlotXY(plot_time, h_voltage, style=Style(line=8, color=Color.black))# Resize the windowp.height = 400 p.width=600# Change the legendp[0].name = "H Mixer Plot"# Change the axis labels...p.xaxis.title.text="Time (hours)" p.yaxis.title.text="Mixer voltage [V]"# ...and add a titlep.title.text="Plot of Mixer Voltages. Start: "+str(prod_date)+\ "End: "+str(end_date)# Now we can also overlay the second voltage trend in blue.p[1]=LayerXY(plot_time, v_voltage, name= "V Mixer Plot", \ style=Style(color=Color.blue))
There are several ways to display the dataframes:
1.-
dd = DisplayDataFrameTask()
dd.dataframe=df[0]
dd.gui = 1
2.-
browse_df = BrowseDfQuery()(gui=1)#feedback dataframes back into your session using:dfs = browse_df.query df = browse_df.selected# feedback dataframes back into your session using:dfs = browse_df.query df = browse_df.selected
3.-
#to pass an existing store this task should reuse:
dfs = AccessDataFrameTask()(store=myStore, .....)
The BrowseHK task is used to display HK data associated with an observation stored in any accessable database. To change to a new database type the name of the database into the local database text field located on the database tab. The fastest way to access data from the database is to lookup data using a specific obsid. If the times in the time selector panel are the same the script will automatically get the begin and end time of the observation and add it to the query string. If the times are different this will override the automatic process and search the database based on those times. The execute button is used to aquire data from the database. The exit button is used to close the application.
# interactive mode (Database: ilt_dm_10 obsid: 268435762)
# interactive mode (Database: ilt_fm_2 obsid: 268435480)
from herschel.hifi.generic.task.BrowseHk import *
hkBrowser = BrowseHk()
hkBrowser()
# This only brings up the browser. Once selections are made, and the
# execute button is pressed, then one can export pk-params(s) in their
# session
#
# resulting_pks = hkBrowser.query // array[TmSourcePacket]!!
# selected_pk = hkBrowser.selected // TableDataset!!
The array of TmSourcePackets (HK data) returned from the query to the database: Array(TmSourcePacket) query [OUTPUT, OPTIONAL, default=None]
To create a TableDataset of HRS HK converted values: hk = HrsGetHKTask()(apid=1028, obsid=268435622, type="All") or alternatevely TableDataset selected [OUTPUT, OPTIONAL, default=None]
To plot a stream of HK data:
#In a new .py file add the lines (call it qlaHkPlot.py) from herschel.hifi.generic.process.hkplot import *DataFlowManager(hkPlot()) Thread.sleep(315360000)#one year in seconds #Then from the terminal window type the followingjylaunch qlaHkPlot.py
To plot and export a time series of HIFI HK data for a selected ObsID
HkPlotter()(obsid = obsid, apid = apid)
HkPlotter()(obsid, apid)
There is an alternative task to plot HK data, which is interactive, in order to analyze and plot WBS functional test: WbsCheckFt .
The On-the-Fly mapping product is a SimpleCube that is generated by the HifiGriddingTask.
Then, double click on HifiGriddingTask in the General Category in HIPE, and set its inputs:
In the future, the HifiGriddingTask will be a part of the pipeline and it will be possible to provide an ObservationContext instead of each individual auxiliary product.
The SimpleCube product can be analyzed with the CubeSpectrumAnalysisToolbox.
Table of Contents
The deconvolution tool is the Level 2 processor to separate the “folded” double sideband (DSB) data inherently produced by the heterodyne process into a single sideband (SSB) result. See the figure below. Fluxes (F) in the DSB baseband are given by:
DSB_IF = g_u*F(LO+IF) + g_l*F(LO-IF)
where LO+/-IF are sky frequencies, and g_l and g_u are sideband gain (imbalance) factors, typically close to 1. The deconvolution is used to reduce WBS Spectral Surveys, which are collections of observations taken at many LO settings so as to constrain the solution. The algorithm finds a SSB solution that best models the observed DSB observations through iterative chi-square minimization (Comito and Schilke 2002). The algorithm automatically runs twice, first with gain factors set to 1.0 for stability, and then a second time, starting with the SSB solution of the first run, this time allowing the gain values to be optimized as well.
Operating the deconvolution tool currently involves three steps, which are described in the next sections. Step 1 (pre-processing the spectra) will eventually disappear as they will be included in either the level 1 pipeline or in the deconvolution tool (steps 2a and 2b) itself.
The level 1 data is fed through a pre-processor, which currently does two things:
Detects and flags spurs: (This will eventually be removed when the spur finder routine is added to the generic HIFI pipeline.) In the meantime, the preprocessor performs the task to satisfy the requirement that bad data are properly flagged before being processed by the deconvolution.
Resamples the fluxes, flags, and weights onto a uniform frequency grid of 0.5 MHz spacing. Flags are interpolated as well, but treated simply as ‘0=good’ and ‘1=bad’. (Further changes to the regridding are planned, and will be applied to weights and flags.) The grid spacing is fixed at 0.5 MHz, but more freedom in the future is planned.
Spectra can be fed into the deconvolution task in HIPE. As demonstrated in the figure of the HIPE Deconvolution GUI below, only 3 user parameters are required:
sampling bin size (typically 0.5 MHz)
tolerance for conversion (typically 0.001)
the maximum number of iterations (typically 200).
All three values are fairly optimized and the user is encouraged to stick with these settings. Pressing ‘run‘ executes the task, and the result is returned as a product.
Alternatively, the deconvolution tool can be run from the command line or in a user jython script. First make sure that the required tasks are imported into your session as follows:
from herschel.hifi.pipeline.product import HifiTimelineProduct from herschel.ia.task import Task from herschel.ia.task import TaskParameter from herschel.hifi.dp.deconvolution import DoDeconvolution
Then the tool can be run either with the default values
decon_result=doDeconvolution(htp=my_htp)
or with user-specified values
decon_result=doDeconvolution(htp=my_htp, bin_size=0,5, max_iterations=200, tolerance=0.001)
A description of the user input variables is given in the previous section (Step 2A).
The product decon_result can be viewed with product viewer.
The single sideband result ssb is a dataset that
can be viewed with the SpectrumExplorer. On the command line, it can be extracted
from the product as follows: ssb=decon_result["ssb"]
The dataset gain can be viewed with dataset inspector. On the command line, it can be extracted
from the product with: gains=decon_result["gain"]
The meta data added to ssb includes the user supplied sampling bin size, number of iterations and the tolerance, as can be seen in the HIPE screenshot below.