[Lilug] MySQL crash and burn (corruption) and trying to recover innodb
Mike A. Leonetti
mikealeonetti at gmail.com
Wed Jun 16 14:05:13 PDT 2010
The MySQL database became corrupt after an unexpected power off. The
backups are impossible, and all I have to work with is the 39 GB
database. MySQL is always crashing in the same spot when I tried to
export with innodb_force_recovery with the values of 4 to 6. I then
tried using both the google innodb-tools and the innoinfo by Steve
Hardy. The innodb-tools doesn't really export much and the innodbinfo
utility gives me:
> Column found for non-existent table 000000000000008E .......?
> Segmentation fault
Below is the error that mysql crashes with. The information in this
database is pretty important. Any ideas?
> 1.
> Jun 15 11:02:54 ender mysqld[8364]: InnoDB: space id 190 did not
> exist in memory. Retrying an open.
> 2.
> Jun 15 11:02:54 ender mysqld[8364]: 100615 11:02:54 InnoDB:
> error: space object of table zarafa/singleinstances,
> 3.
> Jun 15 11:02:54 ender mysqld[8364]: InnoDB: space id 194 did not
> exist in memory. Retrying an open.
> 4.
> Jun 15 11:02:54 ender mysqld[8364]: 100615 11:02:54 InnoDB:
> error: space object of table zarafa/stores,
> 5.
> Jun 15 11:02:54 ender mysqld[8364]: InnoDB: space id 180 did not
> exist in memory. Retrying an open.
> 6.
> Jun 15 11:02:54 ender mysqld[8364]: 100615 11:02:54 InnoDB:
> error: space object of table zarafa/syncs,
> 7.
> Jun 15 11:02:54 ender mysqld[8364]: InnoDB: space id 187 did not
> exist in memory. Retrying an open.
> 8.
> Jun 15 11:02:54 ender mysqld[8364]: 100615 11:02:54 InnoDB:
> error: space object of table zarafa/usergroup_acl,
> 9.
> Jun 15 11:02:54 ender mysqld[8364]: InnoDB: space id 181 did not
> exist in memory. Retrying an open.
> 10.
> Jun 15 11:02:54 ender mysqld[8364]: 100615 11:02:54 InnoDB:
> error: space object of table zarafa/users,
> 11.
> Jun 15 11:02:54 ender mysqld[8364]: InnoDB: space id 182 did not
> exist in memory. Retrying an open.
> 12.
> Jun 15 11:02:54 ender mysqld[8364]: 100615 11:02:54 InnoDB:
> error: space object of table zarafa/versions,
> 13.
> Jun 15 11:02:54 ender mysqld[8364]: InnoDB: space id 188 did not
> exist in memory. Retrying an open.
> 14.
> Jun 15 11:02:55 ender mysqld[8364]: 100615 11:02:55InnoDB:
> Assertion failure in thread 1077102928 in file btr0pcur.c line 403
> 15.
> Jun 15 11:02:55 ender mysqld[8364]: InnoDB: Failing assertion:
> page_is_comp(next_page) == page_is_comp(page)
> 16.
> Jun 15 11:02:55 ender mysqld[8364]: InnoDB: We intentionally
> generate a memory trap.
> 17.
> Jun 15 11:02:55 ender mysqld[8364]: InnoDB: Submit a detailed
> bug report to http://bugs.mysql.com.
> 18.
> Jun 15 11:02:55 ender mysqld[8364]: InnoDB: If you get repeated
> assertion failures or crashes, even
> 19.
> Jun 15 11:02:55 ender mysqld[8364]: InnoDB: immediately after
> the mysqld startup, there may be
> 20.
> Jun 15 11:02:55 ender mysqld[8364]: InnoDB: corruption in the
> InnoDB tablespace. Please refer to
> 21.
> Jun 15 11:02:55 ender mysqld[8364]: InnoDB:
> http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
> 22.
> Jun 15 11:02:55 ender mysqld[8364]: InnoDB: about forcing recovery.
> 23.
> Jun 15 11:02:55 ender mysqld[8364]: 100615 11:02:55 - mysqld got
> signal 11;
> 24.
> Jun 15 11:02:55 ender mysqld[8364]: This could be because you
> hit a bug. It is also possible that this binary
> 25.
> Jun 15 11:02:55 ender mysqld[8364]: or one of the libraries it
> was linked against is corrupt, improperly built,
> 26.
> Jun 15 11:02:55 ender mysqld[8364]: or misconfigured. This error
> can also be caused by malfunctioning hardware.
> 27.
> Jun 15 11:02:55 ender mysqld[8364]: We will try our best to
> scrape up some info that will hopefully help diagnose
> 28.
> Jun 15 11:02:55 ender mysqld[8364]: the problem, but since we
> have already crashed, something is definitely wrong
> 29.
> Jun 15 11:02:55 ender mysqld[8364]: and this may fail.
> 30.
> Jun 15 11:02:55 ender mysqld[8364]:
> 31.
> Jun 15 11:02:55 ender mysqld[8364]: key_buffer_size=16777216
> 32.
> Jun 15 11:02:55 ender mysqld[8364]: read_buffer_size=131072
> 33.
> Jun 15 11:02:55 ender mysqld[8364]: max_used_connections=1
> 34.
> Jun 15 11:02:55 ender mysqld[8364]: max_connections=500
> 35.
> Jun 15 11:02:55 ender mysqld[8364]: threads_connected=1
> 36.
> Jun 15 11:02:55 ender mysqld[8364]: It is possible that mysqld
> could use up to
> 37.
> Jun 15 11:02:55 ender mysqld[8364]: key_buffer_size +
> (read_buffer_size + sort_buffer_size)*max_connections = 1104380 K
> 38.
> Jun 15 11:02:55 ender mysqld[8364]: bytes of memory
> 39.
> Jun 15 11:02:55 ender mysqld[8364]: Hope that's ok; if not,
> decrease some variables in the equation.
> 40.
> Jun 15 11:02:55 ender mysqld[8364]:
> 41.
> Jun 15 11:02:55 ender mysqld[8364]: thd=0x3214210
> 42.
> Jun 15 11:02:55 ender mysqld[8364]: Attempting backtrace. You
> can use the following information to find out
> 43.
> Jun 15 11:02:55 ender mysqld[8364]: where mysqld died. If you
> see no messages after this, something went
> 44.
> Jun 15 11:02:55 ender mysqld[8364]: terribly wrong...
> 45.
> Jun 15 11:02:55 ender mysqld[8364]: Cannot determine thread,
> fp=0x40333fd0, backtrace may not be correct.
> 46.
> Jun 15 11:02:55 ender mysqld[8364]: Stack range sanity check OK,
> backtrace follows:
> 47.
> Jun 15 11:02:55 ender mysqld[8364]: (nil)
> 48.
> Jun 15 11:02:55 ender mysqld[8364]: New value of fp=0x3214210
> failed sanity check, terminating stack trace!
> 49.
> Jun 15 11:02:55 ender mysqld[8364]: Please read
> http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and
> follow instructions on how to resolve the stack trace. Resolved
> 50.
> Jun 15 11:02:55 ender mysqld[8364]: stack trace is much more
> helpful in diagnosing the problem, so please do
> 51.
> Jun 15 11:02:55 ender mysqld[8364]: resolve it
> 52.
> Jun 15 11:02:55 ender mysqld[8364]: Trying to get some variables.
> 53.
> Jun 15 11:02:55 ender mysqld[8364]: Some pointers may be invalid
> and cause the dump to abort...
> 54.
> Jun 15 11:02:55 ender mysqld[8364]: thd->query at 0x324e370 =
> SELECT /*!40001 SQL_NO_CACHE */ * FROM `changes`
> 55.
> Jun 15 11:02:55 ender mysqld[8364]: thd->thread_id=7
> 56.
> Jun 15 11:02:55 ender mysqld[8364]: The manual page at
> http://www.mysql.com/doc/en/Crashing.html contains
> 57.
> Jun 15 11:02:55 ender mysqld[8364]: information that should help
> you find out what is causing the crash.
> 58.
> Jun 15 11:02:55 ender mysqld_safe[8762]: Number of processes
> running now: 0
> 59.
> Jun 15 11:02:55 ender mysqld_safe[8764]: restarted
> 60.
> Jun 15 11:02:55 ender kernel: [ 1872.933119]
> audit(1276614175.648:8): type=1503 operation="capable"
> name="sys_resource" pid=8766 profile="/usr/sbin/mysqld"
> namespace="default"
> 61.
> Jun 15 11:02:56 ender mysqld[8767]: InnoDB: The user has set
> SRV_FORCE_NO_LOG_REDO on
> 62.
> Jun 15 11:02:56 ender mysqld[8767]: InnoDB: Skipping log redo
> 63.
> Jun 15 11:02:56 ender mysqld[8767]: InnoDB: Database page
> corruption on disk or a failed
> 64.
> Jun 15 11:02:56 ender mysqld[8767]: InnoDB: file read of page 4.
> 65.
> Jun 15 11:02:56 ender mysqld[8767]: InnoDB: You may have to
> recover from a backup.
> 66.
> Jun 15 11:02:56 ender mysqld[8767]: 100615 11:02:56 InnoDB: Page
> dump in ascii and hex (16384 bytes):
>
Thanks in advance.
I really should have done backups differently.
--
Mike A. Leonetti
As warm as green tea
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lilug.org/pipermail/lilug-lilug.org/attachments/20100616/6d784b42/attachment-0002.htm>
More information about the Lilug
mailing list