Activity log
Someone
changed Language
to: English
Someone
changed Keywords
to: ["COVID research", "FAIR", "fragalysis", "drug development", "XChem", "PDB", "zenodo", "wp5-ExPaNDS"]
Someone
changed Deliverable
to:
Someone
changed Language
to: English
Someone
changed Keywords
to: ["machine learning", "XFEL", "FAIR", "large dataset", "NeXus", "spectroscopy", "wp5-ExPaNDS"]
Someone
changed Deliverable
to:
antoine
antoine
updated
TELBE data analysis workflow and the PaN training platform UX at 2022-08-25 12:03:25 UTC.
Someone
changed Language
to: English
Someone
changed Keywords
to: ["training platform", "workflow", "TELBE", "FAIR", "data correction", "wp5-ExPaNDS"]
Someone
changed Deliverable
to:
antoine
antoine
updated
Neutron diffraction from Boro-carbon for efficient structural analysis and defect detection at 2022-08-25 12:03:03 UTC.
Someone
changed Language
to: English
Someone
changed Keywords
to: ["McStas", "simulation", "inelastic neutron scattering", "ab-initio", "neutron scattering", "neutron", "neutron diffraction", "defect detection", "structural analysis", "wp5-ExPaNDS"]
Someone
changed Deliverable
to:
Someone
changed Language
to: English
Someone
changed Keywords
to: ["tomography", "pid", "doi", "FAIR", "large dataset", "3Dmagination", "wp5-ExPaNDS"]
Someone
changed Deliverable
to:
Someone
changed Language
to: English
Someone
changed Keywords
to: ["metadata", "open data", "FAIR", "figshare", "binder", "data processing", "wp5-ExPaNDS"]
Someone
changed Deliverable
to:
antoine
antoine
updated
ExPaNDS D5.4 : PaN Training Catalogue Demo Video at 2022-08-25 12:02:26 UTC.
Someone
changed Language
to: English
Someone
changed Keywords
to: ["catalogue", "expands", "deliverable", "wp5-ExPaNDS"]
Someone
changed Deliverable
to:
Someone
changed Language
to: English
Someone
changed Keywords
to: ["expands", "FAIR", "open data", "EOSC", "wp6-ExPaNDS"]
Someone
changed Deliverable
to:
antoine
antoine
modified the workflow diagram at 2022-08-25 11:51:04 UTC.
- Modified 2 nodes: 'Workshop: Hands on session on the PaN ontologies API (Feb 17)', 'PaNET Ontology'
antoine
antoine
modified the workflow diagram at 2022-08-24 13:02:05 UTC.
- Modified 9 nodes: 'Data processing', 'Raw data', 'Reduced data', 'Container', 'Docker (or Singularity) image', 'Cheetah', 'Installation script', 'Local machine', 'HPC cluster'
antoine
antoine
modified the workflow diagram at 2022-08-24 12:54:07 UTC.
- Modified 11 nodes: 'Data processing', 'Raw data', 'Reduced data', 'CrystFEL', 'Container', 'Docker (or Singularity) image', 'Cheetah', 'Slurm (jobs)', 'Installation script', 'Local machine', 'HPC cluster'
antoine
antoine
modified the workflow diagram at 2022-08-24 12:36:49 UTC.
- Modified 8 nodes: 'Experiment', 'Data processing', 'Raw data', 'Reduced data', 'Docker (or Singularity) image', 'Cheetah', 'Installation script', 'Local machine'
antoine
antoine
modified the workflow diagram at 2022-08-24 12:25:06 UTC.
- Modified 2 nodes: 'Introduction', 'Experiment'
antoine
antoine
modified the workflow diagram at 2022-08-24 12:23:19 UTC.
- Modified 7 nodes: 'Introduction', 'Raw data', 'Reduced data', 'Docker (or Singularity) image', 'Cheetah', 'Installation script', 'Local machine'
antoine
antoine
modified the workflow diagram at 2022-08-24 12:21:41 UTC.
- Modified 9 nodes: 'Raw data', 'Reduced data', 'CrystFEL', 'Container', 'Docker (or Singularity) image', 'Cheetah', 'Slurm (jobs)', 'Installation script', 'Local machine'
Someone
changed Description
to: Experiments generate up to 150 TB per day of data saved at the measurement facility. Such large datasets are impractical for users to take home.
Subsequent analysis needs to be performed remotely making it attractive for deployment as a cloud-like use case. Involving EOSC in the analysis and re-use of this data is an appropriate use case.[D4.2]
Serial crystallography is a beamline technique for collecting information on the structure of a protein without growing large protein crystals. Instead, a large number of small protein crystals are held in a pulsed X-ray beam. In a second step, the series of produced images are used to reconstruct a precise 3-D image of the protein structure. Serial crystallography is the preferred technique for obtaining diffraction data of proteins at room temperature, where radiation damage from the X-ray beam starts rapidly. The standard software for analysing serial crystallography is “CrystFEL”.
The proposed workflow was rendered in a standard fashion to allow it to be easily adopted by arbitrary applications. The only requirements are a Docker or Singularity image of the containerised application and a Singularity installation on the target system.
To apply the workflow to a different application other than CrystFEL, only the install script needs to be updated. The required adjustments consist of the image registry, image name and tag as well as the names of the application commands, which are meant to be supported by the containerised deployment. The most recent version of the installation script is provided at DESY GitLab .
Possible todos:
- perhaps put the container in a hpc cluster
- say that the bash installation script builds a Singularity image (not docker) because hpcs prefer those.
todos:
-arrange text more
-add links
Questions:
- probable confusion about which is the target system and which is the host:
Is it correct to say:
- the HOST system is the user's system.
- the TARGET system is the HPC cluster(the TARGET system has to have Singularity installed).
?
- do we meld D4.4 and D4.2 ? Or do we just keep D4.4?
-which study and data to we chose if we do meld D4.2.
(In the WF currently we have the first dataset described in D4.2)
(this is the alternative:
Many other potential datasets at: http://cxidb.org/
Alternative:
http://cxidb.org/id-32.html
Associated publication(s): Sci Data 3, 160021 (2016).
https://doi.org/10.1038/sdata.2016.21
)
this seems to contradict what was said before:
"The container was successfully deployed at DESY’s Maxwell HPC cluster as well as at
MAX IV. In addition to the CrystFEL GUI and commands, Slurm was also accessible from
within the container shell."
meaning slurmis acessible from the container shell?
antoine
antoine
modified the workflow diagram at 2022-08-24 12:02:46 UTC.
- Modified 11 nodes: 'Data processing', 'Raw data', 'Reduced data', 'CrystFEL', 'Container', 'Docker (or Singularity) image', 'Cheetah', 'Slurm (jobs)', 'Installation script', 'Local machine', 'HPC cluster'
antoine
antoine
modified the workflow diagram at 2022-08-24 11:59:56 UTC.
- Modified 13 nodes: 'Experiment', 'Data processing', 'Raw data', 'Reduced data', 'CrystFEL', 'Container', 'Docker (or Singularity) image', 'Publication', 'Cheetah', 'Slurm (jobs)', 'Installation script', 'Local machine', 'HPC cluster'
antoine
antoine
modified the workflow diagram at 2022-08-24 11:56:47 UTC.
- Added 7 nodes: 'CrystFEL', 'Container', 'Docker (or Singularity) image', 'Slurm (jobs)', 'Installation script', 'Local machine', 'HPC cluster'
- Removed 5 nodes: 'container', 'CrystFEL', 'Installation script', 'Slurm (jobs)', 'Docker (or Singularity) image'
- Modified 7 nodes: 'Introduction', 'Experiment', 'Data processing', 'Raw data', 'Reduced data', 'Publication', 'Cheetah'
franz.lang
franz.lang
modified the workflow diagram at 2022-08-24 08:52:05 UTC.
- Modified 2 nodes: 'Data processing', 'Jupyter notebook'
antoine
antoine
updated
Online workshop for the Virtual Infrastructure for Scientific Analysis (VISA) platform at 2022-08-24 08:02:00 UTC.
Someone
changed Content provider
to: ExPaNDS (5)
antoine
antoine
updated
Online workshop for the Virtual Infrastructure for Scientific Analysis (VISA) platform at 2022-08-24 08:01:44 UTC.
Someone
changed Contact
to:
Someone
changed Timezone
to: