|
The database problem
|
|
| Rayden | Date: Monday, 05.12.2011, 16:53 | Message # 1 |
|
Observer
Group: Newbies
Romania
Messages: 6
Status: Offline
| First of all - truly amazing application! 
I'd like to talk a little bit about the database issue in a simulation software like Space Engine, Celestia, etc
Currently SE and Celestia both have the databases stored in the files included in the downloaded archive. The good thing is that this makes the loading time considerably faster than querying a database server. The bad thing is that - if a database inconsistency is found or new planets are added a desynchronization arises between the latest version and the client version.
On the other hand, having the application query the online database server (maintained by the admins) removes these inconsistencies between the latest version and the client version. The downside here is that multiple clients querying the database over and over again for large amounts of data might cause a problem for the server. And the response times, loading times would be somewhat slower.
The way I see it - the best solution would be a combination between the two approaches above.
- have an online database server
- the client application is downloaded with minimal database information
- on application startup the user is given a choice to download the entire database (or maybe parts of it - maybe some are only interested in the local neighbourhood ~200 parsecs)
- on every application startup the user is given a choice to synchronize the database with the online database server (only updated/new information is synchronized to reduce bandwidth)
Why is this necessary? it is not necessary at this moment, when the entire database consists of 700 planets and 107000 stars from the HIP catalog (Tycho is irrelevant here because those 2.6 million stars have high error percentages). From my calculations, the entire database is under 50 MB (Oracle 11gR2 database and including indexes! - the actual data being a lot less).
Compare these 107k stars with the 1-2 billion stars that will be mapped in the GAIA mission.
Rayden
www.3dgalaxymap.com/Galaxy/
|
| |
| |
| SpaceEngineer | Date: Monday, 05.12.2011, 18:04 | Message # 2 |
 Author of Space Engine
Group: Administrators
Russian Federation
Messages: 4800
Status: Offline
| Transfer data to user can be made like transfer of textures are in Google Earth. The database is organized in octree, and this allows it to handle hundreds of billions stars and the interactive choosing of visible ones, like I've done in SpaceEngine.
The galaxy is subdivided into 1000 pc blocks. Each is subdivided into 8 childs 500 pc each. Each of them is subdivided into 8 childs 250 pc each and so on. SpaceEngine has 10-levels of octree. Each octree block has a star array associated with it. The magic is in the range of absolute magnitude for each octree level. The level 0 (1000 pc blocks) handle stars brighter than -5m abs mag, the level 1 handle -5...0m stars, the level 2 handle 0...+2.5m and so on. The step in magnitude is chosen so that each block contains approximately the same number of stars.
Render (or query for downloading) is based on limiting visual magnitude. For example, limiting visual mag = +6m. The Engine passes the octree and chooses the blocks that have visual magnitude of the most brightest block's star is less than +6m. At limiting magnitude +6m and infinite culling radius (sky render mode), the engine needs ~6000 stars in the Sun neighboorhood, at +7m ~50000 stars. This is just several tens of blocks, these will be downloaded (or generated) very quickly. The octree with magnitude culling allows avoiding querying of distant dwarf stars, that are already invisible. Once the blocks are downloaded from server, they may be saved in the disk cache. (For procedural stars, SpaceEngine doesn't use a disk cache, because generating of blocks are much faster than downloading them from disk.) Of course this system makes no sense for HIPPARCOS catalog with only 120k star, but for Gaia it will be neccessary (however, Gaia will be released in 2018-2019, our computers may have terrabytes of RAM by then  )
*
|
| |
| |
| Rayden | Date: Monday, 05.12.2011, 19:08 | Message # 3 |
|
Observer
Group: Newbies
Romania
Messages: 6
Status: Offline
| Yes I believe this can be implemented easily by adding another column for each star in the StarSystem table (that's how I named it :)) - a reference to a visual magnitude type table. Then indexing the table by this newly created column and minimizing the sql query to return only X, Y, Z coordinates, the internal star id (for further queries) and the star type (O, B, A, F, G, K, M) - that's another indexed column.
Now, since the database is already split into 10 parsec sectors - a query can be executed for a range of sectors centered on the user coordinates. It took about 10 seconds to retrieve 50000 stars from the database. The actual number might be lower - I didn't do any actual tracing yet, just timed the data fetch in Toad
About returning the internal star id - very useful for further querying - when the user clicks a certain star - a sql query can return every information on that star that was omitted in the initial star sweep, and it would be pretty instant because it's the primary key.
www.3dgalaxymap.com/Galaxy/
|
| |
| |
| SpaceEngineer | Date: Monday, 05.12.2011, 20:53 | Message # 4 |
|
Author of Space Engine
Group: Administrators
Russian Federation
Messages: 4800
Status: Offline
| No, I talk about the new structure of database, that can easily handle the query for visible stars within any radius from observer - finite or infinite. This is based on octree, split by absolute magnitude of stars.
*
|
| |
| |
| Rayden | Date: Monday, 05.12.2011, 21:24 | Message # 5 |
|
Observer
Group: Newbies
Romania
Messages: 6
Status: Offline
| No, I don't think the database is structured like that
Such a structure would require a full revamp of the database 
www.3dgalaxymap.com/Galaxy/
|
| |
| |
| SpaceEngineer | Date: Monday, 05.12.2011, 23:21 | Message # 6 |
|
Author of Space Engine
Group: Administrators
Russian Federation
Messages: 4800
Status: Offline
| But it allows for zooming the galaxy map.
*
|
| |
| |
ENG
New site
Romania 
Russian Federation