|
SDSS Galaxy collection 2.0 (Updated Sep. 5 2016)
|
|
| ComCypher | Date: Saturday, 13.12.2014, 05:53 | Message # 1 |
|
Space Tourist
Group: SE team
United States
Messages: 21
Status: Offline
|  Region of the Coma Cluster
Presenting the SDSS Galaxy collection. This addon contains 101,640,491 galaxies derived from the Sloan Digital Sky Survey, Phase IV Data Release 13 (http://www.sdss.org/dr13). Galaxy Zoo data (http://www.galaxyzoo.org) was used whenever available to determine the Hubble Tuning Fork classification of each galaxy. In addition, galaxy sizes, distances, luminosities, and orientations are all computed from the source data.
Current version: 2.0 (Sep. 5 2016)
SDSSGalaxyCatalog_GZ_2_0.pak (*Recommended*) (29 MB): Contains a single CSV catalog file containing 639,875 galaxies. These are the galaxies which include Galaxy Zoo data for classification, and tend to be the largest and clearest galaxies in the SDSS dataset.
SDSSGalaxyCatalogs_2_0.pak (*Experimental*) (4.5 GB): Contains 204 CSV catalog files, each containing 500 thousand galaxies. Due to potential limitations with loading large numbers of galaxies, this collection is for experimental use only.
To install, simply copy the PAK file to Space Engine's addons folder. Once in-game, you can search for SDSS galaxies by searching for objects beginning with "SDSS".
 Data coverage
---Known issues---
The number of galaxies that can be loaded at one time may be limited by the software and your PC, which may be noticed when attempting to load more than a couple million galaxies.
Since Galaxy Zoo data is very sparse (only available for less than 1% of the galaxies), for most of the galaxies in the complete collection I had to assign a classification randomly based on known distributions of galaxy types.
Because the SDSS data coverage does not include the entire sky and is sharply defined, it may look odd when traveling in intergalactic space if "auto increase galaxies magnitude" is enabled. I recommend disabling that setting when using this mod.
Not an issue per se, but the number of galaxies in this mod is about 100 million less than what exists in the SDSS database. This is due to data cleansing that I needed to perform to remove galaxies with incomplete or faulty data.
In addition to millions of minor galaxies, the SDSS database also contains "famous" galaxies which can be found in other catalogs such as NGC. However, SE already includes many of those galaxies in its galaxies10k.sc file. This means that duplicates may be observed in-game where the SDSS galaxy overlaps the default SE galaxy. I haven't made any effort to identify or remove these duplicate galaxies in the current release.
---Information for developers---
This is an open mod so I'm providing all of the information that was used to process the data, in case anyone wants to extend or improve upon it themselves.
SDSS-IV-DR13-GalaxySourceData.zip (3.7 GB / 18.7 GB uncompressed): This zip contains a single large CSV file which is the output of the SQL resultset on the SDSS database.
SDSSGalaxyQuery.sql: The SQL query that was used to pull the required columns from the database.
SDSSGalaxyWriter.py: The Python script that was used to transform the source data into SE catalog files.
SDSS navigation tool: This is a handy viewer for visually inspecting the galaxies in the SDSS database.
CasJobs workbench: It is possible to create an account here and run queries against the SDSS SQL Server database, although the amount of data that can be downloaded may be limited (a member of the science team assisted me with the full extraction).
 600,000+ galaxies (magnitude artificially increased)
Edited by ComCypher - Monday, 05.09.2016, 07:15 |
| |
| |
| DeathStar | Date: Saturday, 13.12.2014, 12:16 | Message # 2 |
|
Pioneer
Group: Users
Croatia
Messages: 515
Status: Offline
| Wow.... just .... wow. Just to let you know(you probably do already) there have been changes to the catalog loading system in 0.9.7.2 so SpaceEngineer said that it might be possible to load the entire SDSS database in a reasonable time. Maybe you should contact him so that he can try and include this catalog.
|
| |
| |
| SpaceEngineer | Date: Saturday, 13.12.2014, 14:21 | Message # 3 |
 Author of Space Engine
Group: Administrators
Russian Federation
Messages: 4800
Status: Offline
| ComCypher, thanks for such a great addon!
The first mistake: comment symbols is SE are "//", not "\\". Version 0.972 have an error check, and it immediately wrote an errors in the log.
Second, you can significantly reduce the file size (and loading time) be deleting unnecessary spaces between the parameter and its value (you may leave a single tabulation), remove unnecessary leading "+" and trailing ".0", between opening and closing round bracket in the quaternion, also in the quaternion you can remove comas (they are not necessarily) and reduce accuracy to 6-7 digits. Example:
Code
Galaxy "SDSS 1237648722848776804"
{
Type "Irr"
RA 13.1323
Dec 1.17823
Dist 243708336
Radius 2276.75089
AbsMagn -20.64
Quat (0.2530524 0.9486663 -0.1538515 -0.1110235)
}
This edited entry have 186 characters, compared to 237 of original entry is almost 30% win.
Also, if a galaxy have no orientation data, you may skip the Quat parameter - SE will generate it procedurally.
In future I will implement a binary file format for such huge catalogs, with loading of the tiles on-the-fly, like planetary textures (or generation of procedural galaxies). But for now the text catalog should be compressed as more as possible.
Anyway, SE 0.971 could not load all 53 catalogs for a obvious cause: their total size is 23 Gb, but SE can't load more than 4 Gb because it is a 32-bit app. Even if it would be a 64-bit app, I think only few people would have computers capable to load 23 Gb into RAM. Current 0.972 implementation even can't load a single catalog, because of limitation in the std::vector class (it is used in new accelerated script loader). So this great addon should wait implementing of the binary tiled catalog. Now I am forced to implement it for the next 0.973 version 
I have a suggestion now. What if limit the catalog size by 500,000 galaxies, selecting them by some principle (the most big, the most luminous, belonged to certain survey, belonged to a certain "2d" slice, etc). SE could load 500k galaxies (I just tested it - perfect performance, no problems in the map mode). This "filtered" catalog could be used with 0.971 and 0.972 until I implement the binary catalog.
Regarding performance. You should use a parameters to set up the ocetree size and depth. Put something like this in the beginning of the catalog script:
Code
// Octree parameters
OctreeSize ( 1.0e9 1.0e9 1.0e9 )
OctreePos ( 2.0e7 3.0e8 2.0e8 )
OctreeDepth 5
OctreeSize is s span of the octree in parsecs. it should be choosen in that way, so 95% of galaxies would be inside it. The rest 5% (25k gakaxies for a 500k catalog) would be placed in a special resized octree nodes. This would get the best performance.
OctreePos is a location of the center of the octree. It should not be zero, otherwise the MilkyWay could be clip out at some camera positions. Ie is should be located somewhere near the Milky Way, but in the intergalactic space.
OctreeDepth is a depth level of the octree. The good value for a 500k catalog is 5-6.
Note that SE takes octree parameters only from the first file, specified in the GalaxiesCatalogs in the universe.cfg. So your huge addons should be specified *before* the galaxies10k.sc, otherwise SE will use parameters for a small octree form the 10k catalog, resulting in extremely poor performance.
You should experiment with these parameters to find a best performance. Disable procedural galaxies and all stars, planets, clusters and nebulae (so only catalog galaxies left). Disable VSync in the display menu to see the real FPS. Find several test locations and save them in the F6 menu. Locations could be inside the Milky Way, somewhere inside the catalog, and few gigaparsecs outside it. Measure the FPS in those locations at certain galaxy magnitude level (F7 menu). Then change the octree parameters in the script, restart SE and measure again.
|
| |
| |
| ComCypher | Date: Saturday, 13.12.2014, 15:13 | Message # 4 |
|
Space Tourist
Group: SE team
United States
Messages: 21
Status: Offline
| Those are great suggestions SpaceEngineer, thanks (I'm particularly embarrassed by the incorrect comment slashes  ). I will definitely get those things fixed and optimized in the next release.
I also like your idea to create an additional "light" catalog of 500k galaxies. I can probably create a few based on size, luminosity, or Galaxy Zoo. Back to work  .
|
| |
| |
| Salvo | Date: Saturday, 13.12.2014, 20:40 | Message # 5 |
 Star Engineer
Group: Local Moderators
Italy
Messages: 1400
Status: Offline
| If my connection was better, I would definitely download this 
Good job!
The universe is not required to be in perfect harmony with human ambition.
CPU: Intel Core i7 4770 GPU: ASUS Radeon R9 270 RAM: 8 GBs
(still don't know why everyone is doing this...)
|
| |
| |
| ComCypher | Date: Saturday, 20.12.2014, 01:34 | Message # 6 |
|
Space Tourist
Group: SE team
United States
Messages: 21
Status: Offline
| Changelog: 1.1
Created a "light" catalog containing 665,781 galaxies with Galaxy Zoo data, which should look and perform well on most PCs.
Octree settings have been computed and included in each catalog file which should help improve performance.
Reduced the number of galaxies per catalog from 2 million to 500 thousand to reduce the memory footprint of each file.
Corrected orientation calculations for elliptical galaxies.
Various catalog formatting fixes and optimizations to reduce file size. The size of the uncompressed distribution is now about 20% smaller than before.
|
| |
| |
| SpaceEngineer | Date: Monday, 22.12.2014, 09:43 | Message # 7 |
|
Author of Space Engine
Group: Administrators
Russian Federation
Messages: 4800
Status: Offline
| When 0.972 will be released, you can make a pak files for this addon, and distribute it in that form. Pak files are zip archives, compression for text files is usually very good, so this catalogs will have a huge benefit from that.
Update: I just copied the SDSSGalaxyCatalog_GZ_1_1.zip into catalogs/galaxies folder and renamed it into .pak, and it worked)
|
| |
| |
| isdebesl | Date: Tuesday, 23.12.2014, 00:40 | Message # 8 |
 Space Pilot
Group: Users
Indonesia
Messages: 87
Status: Offline
| the link doesn't work
|
| |
| |
| ComCypher | Date: Tuesday, 23.12.2014, 00:57 | Message # 9 |
|
Space Tourist
Group: SE team
United States
Messages: 21
Status: Offline
| Quote SpaceEngineer (  ) When 0.972 will be released, you can make a pak files for this addon, and distribute it in that form.
Yes indeed, already on my to-do list . As soon as 0.972 is released I'll put up a pak link and update the instructions.
Quote isdebesl ( ) the link doesn't work
Can you elaborate on which link is giving you trouble? I just tried both and they seemed fine.
Edited by ComCypher - Tuesday, 23.12.2014, 00:58 |
| |
| |
| isdebesl | Date: Tuesday, 23.12.2014, 02:00 | Message # 10 |
|
Space Pilot
Group: Users
Indonesia
Messages: 87
Status: Offline
| I'm clicking the recommended one, it load very slow (not because slow connection), and its said webpage is not available, im check the short url checker, and the result is this
 Added (23.12.2014, 01:00)
---------------------------------------------
plz halp
|
| |
| |
| ComCypher | Date: Tuesday, 23.12.2014, 03:03 | Message # 11 |
|
Space Tourist
Group: SE team
United States
Messages: 21
Status: Offline
| I'm not sure if that URL checker is reliable. Here is the actual long link, give this one a try:
*EDIT: Redundant link removed.*
Edited by ComCypher - Tuesday, 23.12.2014, 19:14 |
| |
| |
| isdebesl | Date: Tuesday, 23.12.2014, 11:37 | Message # 12 |
|
Space Pilot
Group: Users
Indonesia
Messages: 87
Status: Offline
| The link's work! thx, actually i'm download it at firefox, it doesn't work on google chrome, idk why, but thx alot! Added (23.12.2014, 10:37)
---------------------------------------------
Oh ya, i have a suggestion, what if SDSS Stars collection?? it would be awesome and make this game more fun to explore!! how 'bout that?
|
| |
| |
| ComCypher | Date: Tuesday, 23.12.2014, 19:12 | Message # 13 |
|
Space Tourist
Group: SE team
United States
Messages: 21
Status: Offline
| Glad to hear that. Also I have thought about processing the SDSS stars, but obviously stars have very different properties from galaxies so I would have to develop a brand new process to handle those. I haven't done much research it that yet but I might get around to it at some point.
|
| |
| |
| SpaceEngineer | Date: Tuesday, 23.12.2014, 22:10 | Message # 14 |
|
Author of Space Engine
Group: Administrators
Russian Federation
Messages: 4800
Status: Offline
| I can point on a obvious issue, that can make impossible to use SDSS stars catalog in SE. Are they have a distance measured? I guess no.
|
| |
| |
| isdebesl | Date: Wednesday, 24.12.2014, 03:52 | Message # 15 |
|
Space Pilot
Group: Users
Indonesia
Messages: 87
Status: Offline
| Oh, sorry, i didn't know that Added (24.12.2014, 02:52)
---------------------------------------------
Quasars: http://das.sdss.org/va/qso_properties_dr7/data/catalogs/
Stripe 82 Standard Star Catalog: http://www.astro.washington.edu/users....82.html
UCAC3 Catalog (maybe? idk): http://sourceforge.net/projects/skychart/files/2-catalogs/Stars/UCAC3/
|
| |
| |
ENG
New site
United States 
Croatia 
Russian Federation 
Italy 
Indonesia