­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ 3 i.@sddlZddlZddlZddlZddlZddlZddlZddlmZm Z ddl m Z ddl m Z ddlmZddlmZddlmZdd lTdd lmZdd lmZmZmZmZmZdd lmZdd lmZddlmZddlmZddlm Z m!Z!ddlm"Z"ddl#m$Z$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1ddl2m3Z3m4Z4e(e5Z'dZ6e7e6Z8dZ9e7e9e8kst:dZ;dZee e?Z@GdddZAGdddZBeBjCjDd kst:dS)!N)hexlify unhexlify) defaultdict) ConfigParser)datetime)partial)islice)*)NSIndex)ErrorErrorWithTracebackIntegrityErrorformat_file_sizeparse_file_size)Location)ProgressIndicatorPercent) bin_to_hex)hostname_is_unique) secure_erase safe_unlink)msgpack)Lock LockError LockErrorT) create_logger)LRUCache)SaveFileSyncFilesync_dir safe_fadvise)crc32)IntegrityCheckedFileFileIntegrityErrorsBORG_SEGsATTICSEGc@sbeZdZdZGdddeZGdddeZGdddeZGdd d eZGd d d eZ Gd d d eZ GdddeZ Gddde Z Gddde ZGdddeZGdddeZd{ddZddZdd Zd!d"Zd#d$Zed%d&Zed'd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4Zd5d6Z d7d8Z!d9d:Z"d;d<Z#d=d>Z$d?d@Z%dAdBZ&d|dCdDZ'dEdFZ(d}dGdHZ)dIdJZ*d~dKdLZ+ddMdNZ,dOdPZ-dQdRZ.dSdTZ/dUdVZ0dWdXZ1ddYdZZ2d[d\Z3dd]d^Z4d_d`Z5dadbZ6dcddZ7dedfZ8dgdhZ9ddidjZ:ddkdlZ;dmdnZddsdtZ?dudvZ@ddwdxZAdydzZBdS) Repositorya Filesystem based transactional key value store Transactionality is achieved by using a log (aka journal) to record changes. The log is a series of numbered files called segments. Each segment is a series of log entries. The segment number together with the offset of each entry relative to its segment start establishes an ordering of the log entries. This is the "definition" of time for the purposes of the log. Log entries are either PUT, DELETE or COMMIT. A COMMIT is always the final log entry in a segment and marks all data from the beginning of the log until the segment ending with the COMMIT as committed and consistent. The segment number of a segment ending with a COMMIT is called the transaction ID of that commit, and a segment ending with a COMMIT is called committed. When reading from a repository it is first checked whether the last segment is committed. If it is not, then all segments after the last committed segment are deleted; they contain log entries whose consistency is not established by a COMMIT. Note that the COMMIT can't establish consistency by itself, but only manages to do so with proper support from the platform (including the hardware). See platform.base.SyncFile for details. A PUT inserts a key-value pair. The value is stored in the log entry, hence the repository implements full data logging, meaning that all data is consistent, not just metadata (which is common in file systems). A DELETE marks a key as deleted. For a given key only the last entry regarding the key, which is called current (all other entries are called superseded), is relevant: If there is no entry or the last entry is a DELETE then the key does not exist. Otherwise the last PUT defines the value of the key. By superseding a PUT (with either another PUT or a DELETE) the log entry becomes obsolete. A segment containing such obsolete entries is called sparse, while a segment containing no such entries is called compact. Sparse segments can be compacted and thereby disk space freed. This destroys the transaction for which the superseded entries where current. On disk layout: dir/README dir/config dir/data// dir/index.X dir/hints.X File system interaction ----------------------- LoggedIO generally tries to rely on common behaviours across transactional file systems. Segments that are deleted are truncated first, which avoids problems if the FS needs to allocate space to delete the dirent of the segment. This mostly affects CoW file systems, traditional journaling file systems have a fairly good grip on this problem. Note that deletion, i.e. unlink(2), is atomic on every file system that uses inode reference counts, which includes pretty much all of them. To remove a dirent the inodes refcount has to be decreased, but you can't decrease the refcount before removing the dirent nor can you decrease the refcount after removing the dirent. File systems solve this with a lock, and by ensuring it all stays within the same FS transaction. Truncation is generally not atomic in itself, and combining truncate(2) and unlink(2) is of course never guaranteed to be atomic. Truncation in a classic extent-based FS is done in roughly two phases, first the extents are removed then the inode is updated. (In practice this is of course way more complex). LoggedIO gracefully handles truncate/unlink splits as long as the truncate resulted in a zero length file. Zero length segments are considered to not exist, while LoggedIO.cleanup() will still get rid of them. c@seZdZdZdS)zRepository.DoesNotExistzRepository {} does not exist.N)__name__ __module__ __qualname____doc__r+r+"/usr/lib64/python3.6/repository.py DoesNotExist{sr-c@seZdZdZdS)zRepository.AlreadyExistsz"A repository already exists at {}.N)r'r(r)r*r+r+r+r, AlreadyExists~sr.c@seZdZdZdS)zRepository.PathAlreadyExistsz!There is already something at {}.N)r'r(r)r*r+r+r+r,PathAlreadyExistssr/c@seZdZdZdS)z!Repository.ParentPathDoesNotExistz:The parent path of the repo directory [{}] does not exist.N)r'r(r)r*r+r+r+r,ParentPathDoesNotExistsr0c@seZdZdZdS)zRepository.InvalidRepositoryz0{} is not a valid repository. Check repo config.N)r'r(r)r*r+r+r+r,InvalidRepositorysr1c@seZdZdZdS)z"Repository.InvalidRepositoryConfigz?{} does not have a valid configuration. Check repo config [{}].N)r'r(r)r*r+r+r+r,InvalidRepositoryConfigsr2c@seZdZdZdS)zRepository.AtticRepositoryz8Attic repository detected. Please run "borg upgrade {}".N)r'r(r)r*r+r+r+r,AtticRepositorysr3c@seZdZdZdS)zRepository.CheckNeededz3Inconsistency detected. Please run "borg check {}".N)r'r(r)r*r+r+r+r, CheckNeededsr4cs eZdZdZfddZZS)zRepository.ObjectNotFoundz.Object with key {} not found in repository {}.cs$t|trt|}tj||dS)N) isinstancebytesrsuper__init__)selfidZrepo) __class__r+r,r8s z"Repository.ObjectNotFound.__init__)r'r(r)r*r8 __classcell__r+r+)r;r,ObjectNotFoundsr=c@seZdZdZdS)z%Repository.InsufficientFreeSpaceErrorzNInsufficient free space to complete transaction (required: {}, available: {}).N)r'r(r)r*r+r+r+r,InsufficientFreeSpaceErrorsr>c@seZdZdZdS)zRepository.StorageQuotaExceededzJThe storage quota ({}) has been exceeded ({}). Try deleting some archives.N)r'r(r)r*r+r+r+r,StorageQuotaExceededsr?FNTc Cstjj||_td|j|_d|_d|_d|_i|_d|_ ||_ ||_ ||_ d|_ ||_||_||_d|_d|_||_| |_dS)Nz file://%sFr)ospathabspathrZ _locationiolockindex shadow_index _active_txn lock_waitdo_lock do_createcreated exclusive append_only storage_quotastorage_quota_usetransaction_doomedcheck_segment_magicmake_parent_dirs) r9rAcreaterLrHrDrMrNrQrRr+r+r,r8s$zRepository.__init__cCs|jr|jdstddS)NFz&cleanup happened in Repository.__del__)rDcloseAssertionError)r9r+r+r,__del__szRepository.__del__cCsd|jj|jfS)Nz<%s %s>)r;r'rA)r9r+r+r,__repr__szRepository.__repr__cCs@|jrd|_|j|jd|_|j|jt|j|j|jd|S)NFT)rHrD) rJrSrArKopenboolrLrHrI)r9r+r+r, __enter__s  zRepository.__enter__cCsR|dk rF|tko|jtjk}|jr6|r6tjdd}nd}|j|d|jdS)NzGNo space left on device, cleaning up partial transaction to free space.TF)cleanup)OSErrorerrnoZENOSPCrGloggerwarning _rollbackrT)r9exc_typeZexc_valZexc_tbZno_space_left_on_devicer[r+r+r,__exit__s   zRepository.__exit__cCs t|jS)N)rr:)r9r+r+r,id_strszRepository.id_strcCsTy:ttjj|dd}|jd}d|kp0d|kSQRXWntk rNdSXdS)z;Check whether there is already a Borg repository at *path*.READMErbdsBorg Backup repositorysBorg repositoryNF)rXr@rAjoinreadr\)rAfdZ readme_headr+r+r, is_repositorys  zRepository.is_repositoryc Csytj|}Wntk r"Yn8X|j|r8|j|tj|j sPtj|rZ|j|x>|}tj j tj j |tj }||krP|j|r\|j|q\WdS)a Raise an exception if a repository already exists at *path* or any parent directory. Checking parent directories is done for two reasons: (1) It's just a weird thing to do, and usually not intended. A Borg using the "parent" repository may be confused, or we may accidentally put stuff into the "data/" or "data//" directories. (2) When implementing repository quotas (which we currently don't), it's important to prohibit folks from creating quota-free repositories. Since no one can create a repository within another repository, user's can only use the quota'd repository, when their --restrict-to-path points at the user's repository. N) r@statFileNotFoundErrorrjr.S_ISDIRst_modelistdirr/rArBrgpardir)r9rAstZ previous_pathr+r+r,check_can_create_repositorys     z&Repository.check_can_create_repositorycCsv|j||jr.tjj|tj}tj|ddtjj|sxytj|Wn.t k rv}z|j ||WYdd}~XnXt tjj|dd}|j t WdQRXtjtjj|dtdd}|jd|jdd d |jdd tt|jdd tt|jdd tt|j|jr2|jddt|jn|jddd|jddd|jddttjd|j||dS)z0Create a new empty repository at `path` T)exist_okNrdwdata) interpolation repositoryversion1segments_per_dirmax_segment_sizerMrN0additional_free_spacer: )rrrRr@rArgrpmakedirsexistsmkdirrlr0rXwriteZREPOSITORY_READMErZ add_sectionsetstrZDEFAULT_SEGMENTS_PER_DIRZDEFAULT_MAX_SEGMENT_SIZEintrMrNrurandom save_config)r9rA parent_patherrriconfigr+r+r,rS s0    zRepository.createc CsTtjj|d}tjj|d}tjj|r>tjdt|ddtjj|rd}ytj||Wnntk r}z6|j t j t j t j t j t jt jfkrtj|nWYdd}~Xntk rtj|YnXy"t|}|j|WdQRXWnDtk r4}z&|jrtjd|j|jfWYdd}~XnXtjj|rPt|dddS)Nrz config.oldz=Old config file not securely erased on previous config updateT)Zavoid_collateral_damagezFailed to securely erase old repository config file (hardlinks not supported). Old repokey data, if any, might persist on physical storage.zT%s: Failed writing to '%s'. This is expected when working on read-only repositories.)r@rArgisfiler^r_rlinkr\r]ZEMLINKZENOSYSZEPERMZEACCESZENOTSUPZEIOAttributeErrorrrPermissionErrorrIstrerrorfilename)r9rArZ config_pathZold_config_pathZlink_error_msgerir+r+r,r*s2    "  "zRepository.save_configcCs8|js t|jd}|jjdd||j|j|jdS)Nzutf-8rwkey)rrUdecoderrrA)r9keydatar+r+r,save_keyMs  zRepository.save_keycCs |jjddddj}|jdS)Nrwr)fallbackzutf-8)rgetstripencode)r9rr+r+r,load_keySszRepository.load_keycCsp|jr|jj rtdtjj|jd}y,t|d}tj t |j ddSQRXWnt k rjdSXdS)Nz-bug in code, exclusive lock should exist herenoncerbig) byteorder) rIrDgot_exclusive_lockrUr@rArgrXr from_bytesrrhrl)r9 nonce_pathrir+r+r,get_free_nonceXs  zRepository.get_free_noncecCs|jr|jj rtd|j|kr.tdtjj|jd}y4t |dd}|j t |j dddWdQRXWn@t k r}z$|jrtjd |j|jfWYdd}~XnXdS) Nz-bug in code, exclusive lock should exist herez6nonce space reservation with mismatched previous staterF)binaryr)rzT%s: Failed writing to '%s'. This is expected when working on read-only repositories.)rIrDrrUr Exceptionr@rArgrrrto_bytesrr^r_rr)r9Znext_unreservedZ start_noncerrirr+r+r,commit_nonce_reservationcs &z#Repository.commit_nonce_reservationcCsB|jrt|jd|jtjtjj|jdtj|jdS)z.Destroy the repository at `self.path` z is in append-only moderN) rM ValueErrorrArTr@removergshutilZrmtree)r9r+r+r,destroyts zRepository.destroycs2tfddtjjD}|r*|dSdSdS)Nc3sT|]L}|jdr|ddjrtjtjjj|jdkrt|ddVqdS)zindex.Nr) startswithisdigitr@rkrArgst_sizer).0fn)r9r+r, ~sz6Repository.get_index_transaction_id..r )sortedr@rorA)r9indicesr+)r9r,get_index_transaction_id}s  z#Repository.get_index_transaction_idcCsh|j}|jj}|dk r6|dkr6d|j}|j|||krd|dk rT||krTd}n|}|j||dS)Nz,%s" - although likely this is "beyond repair)rrCget_segments_transaction_idrAr4replay_segments)r9index_transaction_idsegments_transaction_idmsgZ replay_fromr+r+r,check_transactions   zRepository.check_transactioncCs|j|jS)N)rr)r9r+r+r,get_transaction_idszRepository.get_transaction_idcCsttjj|jdjdS)NrD)rr@rArg break_lock)r9r+r+r,rszRepository.break_lockcCs|jdk r|jj||dS)N)rD migrate_lock)r9Zold_idZnew_idr+r+r,rs zRepository.migrate_lockc CsB||_ytj|}Wntk r2|j|YnXtj|jsJ|j||rrttjj |d||t dj |_ nd|_ t dd|_y0ttjj |jd}|jj|WdQRXWn(tk r|j|j|jYnXd|jjkr|j|j|d|jjdd}|dkr2|j|j|d |t|jjdd |_|jtkrj|j|j|d t|jjdd |_t|jjdd dd|_|jp|jjdddd|_|jdkrt|jjdddd|_t|jjddj|_ t!|j|j|j|_"|j#r>|j"j$}|dk r>|j"j%|t&kr>|j|j'|dS)NrD)ZtimeoutZkill_stale_locks)rvrrwzno repository section foundrxr z;repository version %d is not supported by this borg versionr{zmax_segment_size >= %drzr}r)rrMFrNr:)(rAr@rkrlr-rmrnr1rrgracquirerDrrrXZ read_filerTZsectionsr2Zgetintrrr{ZMAX_SEGMENT_SIZE_LIMITrzr}rMZ getbooleanrNrrr:LoggedIOrCrQget_latest_segmentget_segment_magic ATTIC_MAGICr3) r9rArLrHrDrqriZ repo_versionsegmentr+r+r,rXsT  $       zRepository.opencCs0|jr,|jr|jjd|_|jjd|_dS)N)rDrCrTrelease)r9r+r+r,rTs   zRepository.closecCsT|jr|j}|j||j|j|jj|js@|j|j|jdS)zCommit transaction N) rProllbackcheck_free_spacelog_storage_quotarC write_commitrMcompact_segments write_index)r9 save_spaceZ exceptionr+r+r,commits zRepository.commitcCsd|}tjj|j|}y$t|d}tj|}WdQRXWntk rPdSX|jddkrxtj d|jd|dS||j S)Nz integrity.%dresversionr$z'Unknown integrity data version %r in %s) r@rArgrXrunpackrlrr^r_r)r9transaction_idrintegrity_fileZintegrity_pathri integrityr+r+r,_read_integritys zRepository._read_integritycCs|dkrtStjj|jd|}|j|d}y$t|d|d }tj|SQRXWnbttt fk r}z@t j d|tj ||s|j |j|j|j|jSd}~XnXdS)Nzindex.%dsindexF)rintegrity_datazARepository index missing or corrupted, trying to recover from: %s)r r@rArgrr"rhrr\r#r^r_unlink prepare_txnrr open_index)r9r auto_recover index_pathrriexcr+r+r,rs   zRepository.open_indexc (Csd|_|jrZ|jj rZ|jdk r*tdy|jjWn ttfk rXd|_YnX|j sj|dkry|j |dd|_ WnLt t t fk r}z*tjd||j|j |dd|_ WYdd}~XnX|dkri|_t|_d|_|jjn|r|jj|tjj|jd|}tjj|jd|}|j|d }y(t|d|d }tj|}WdQRXWnltj tj!t"t fk r} zBtjd | t#| t"stj$|tj$||j|j%|dSd} ~ XnX|d d kr>tj&d||d|_t|_d|_x,t'|dD]} tj&d| |j(| qWtj&dnF|d dkr^t d|d n&|d|_t|d|_|j)dd|_|j*x@|jj+D]2\} } x&t,| D]} | |kr| j-| qWqWdS)NTz-bug in code, exclusive lock should exist hereF)rz9Checking repository transaction due to previous error: %srzhints.%dzindex.%dshints)rrzARepository hints file missing or corrupted, trying to recover: %ssversionr zUpgrading from v1 hints.%dssegmentsscompactz%Rebuilding sparse info for segment %dzUpgrade to v2 hints completer$zUnknown hints file version: %dsstorage_quota_use).rGrIrDrrLrUZupgraderrrErrr\r#r^r_rsegments FreeSpacecompactrOrFclearrCr[r@rArgrr"rrZUnpackExceptionZ ExtraDatarlr5rrdebugr_rebuild_sparserritemslistr) r9r do_cleanuprZ hints_pathrrrihintsrrrZshadowed_segmentsr+r+r,rsp  "             zRepository.prepare_txnc!Csdd}dd}d|j|j|jd}ddi}|jj}|dk sBt|jrttj j |j dd $}t d |t j jtf|d WdQRXd |}tj j |j |}t|d |dd} tj|| || WdQRX| j|d<d|} tj j |j | } t| d | dd} |jj| || WdQRX| j|d<d|} tj j |j | } t| d d} tj|| || WdQRX|| t|j |||| t|j d|}xLtj|j D]<}|jds̐q|j|rܐqtjtj j |j |qWd|_dS)NcSs|jtj|jdS)N)flushr@fsyncfileno)rir+r+r,flush_and_syncWsz.Repository.write_index..flush_and_synccSstj|d|dS)Nz.tmp)r@rename)filer+r+r, rename_tmp[sz*Repository.write_index..rename_tmpr$)sversionssegmentsscompactsstorage_quota_usesversionZ transactionsaztransaction %d, UTC time %s)rzhints.%dz.tmpT)rrshintszindex.%dsindexz integrity.%dwbz.%dindex.hints. integrity.)rrr)rrrOrCrrUrMrXr@rArgprintrZutcnowZstrftimeZ ISO_FORMATr"rpackrrErrrorendswithr)r9rrrrrlogZ hints_nameZ hints_fileriZ index_nameZ index_fileZintegrity_namerZcurrentnamer+r+r,rVsV   $         zRepository.write_indexc Cs|jjd}t|jdt|jdd}||7}||j7}|js|jt}t|jdkrd}xD|jj D]6\}}y||j j ||7}Wqnt k rYqnXqnWt jdt||}t jd|||7}n||7}ytj|j}Wn4tk r}zt jdt|dSd}~XnX|j|j} t jd j|| | |kr|jrdt jd |jn |jd d t|} t| } |j| | dS) zJPre-commit check for sufficient free space to actually perform the commit. irzAcheck_free_space: few segments, not requiring a full free segmentzBcheck_free_space: calculated working space for compact as %d bytesz.Failed to check free space before committing: Nz2check_free_space: required bytes {}, free bytes {}z@Not enough free space to initialize repository at this location.T)r[)rEsizelenrrr}rMr{MAX_OBJECT_SIZErrC segment_sizerlr^rminr@statvfsrAr\r_rf_bavailf_frsizeformatrKerrorrr`rr>) r9Zrequired_free_spaceZ hints_sizeZfull_segment_sizeZcompact_working_spacerZfreeZst_vfsZos_errorZ free_spaceZformatted_requiredZformatted_freer+r+r,rsB              zRepository.check_free_spacecCs$|jr tjdt|jt|jdS)Nz!Storage quota: %s out of %s used.)rNr^inforrO)r9r+r+r,rszRepository.log_storage_quotacsjs dSj}j}j}gtddfdd }jdttjddd d }xtjj D]\}j j sj d j=|j qnj j}d ||}|d jkr|d|krjd|d||j qn|jdjd||d|xj jddD]\} } } } | tkrHq,jj| } | | fk}| tkr|ryj j| | dd\}} Wn0tjk r|j j| | \}} YnX|| fj| <|j|d||d7<|d8<q,| tkrZ| rZyj| jWnttfk r:YnXjt| j jj8_n| tkr,| r,| jkpt fddj| D}|dkp|k}|s|ryj j!| dd\}}Wn.tjk r|j j!| \}}YnXj||7<|j|dnj| s,j| =q,W|dksJt"dj#|j qnW|j$|ddj}j%dt&||jddS)zBCompact sparse segments by copying data into new segments Nzborg.debug.compact_segmentsTcsvjj|d}jd|rdnd|xHD]@}jd|jj|}|dksVtdjj|j|=q*WgdS)N) intermediatez+complete_xfer: wrote %scommit at segment %dz intermediate rz)complete_xfer: deleting unused segment %drz.complete_xferzcompaction started.zCompacting segments %3.0f%%r zrepository.compact_segments)totalrstepmsgidz3segment %d not found, but listed in compaction datag?g?g333333?z>not compacting segment %d (maybe freeable: %2.2f%% [%d bytes])gY@rzNcompacting segment %d with usage count %d (maybe freeable: %2.2f%% [%d bytes])) include_data) raise_fullc3s|]}|kVqdS)Nr+)rZshadowed)rr+r,r'sz.Repository.compact_segments..z.zReplaying segments %3.0f%%zrepository.replay_segments)r rr ) rLrsumrCsegment_iteratorr enumeraterr _update_indexrrr) r9rrZremember_exclusive segment_countrirrobjectsr+r+r,ras&   zRepository.replay_segmentsc Csd|j|<xN|D]D\}}}}|tkry6|j|\}} |j||7<|j|d8<Wntk rrYnX||f|j|<|j|d7<|j|7_q|tkr y|jj|\}}Wntk rYnHX|jj |rX|j|d8<|jj |||dd}|j||7<q|t kr.qqdj ||} |dkrP|j | q|| qW|j|dkr~|jj||j|<dS)z2some code shared between replay_segments and checkrr F) read_datazUnexpected tag {} in segment {}N)rrrErrrOrrrCrrhrrr4r) r9rr(Zreportrrr rsr!rr+r+r,r%xs:       zRepository._update_indexc Csy|jj|}Wn tk r0|jj|dSX|j|dkrN||j|<dSd|j|<xl|jj|ddD]X\}}}}|tkr|jj |d||fkr|j||7<qj|t krj|j||7<qjWdS) zNRebuild sparse bytes count for a single segment relative to the current index.NrF)r)r rr)rr) rCrrlrrrrrrErr)r9rrrrr rr+r+r,rs   zRepository._rebuild_sparsec!s|jr|rt|jddfdd}tjd|j s>ty"|j}|j|}tj d|Wn<t k r}z |j j }d}tj d|WYdd}~XnX|dkrtj d |j }|dkrtj d |j j}|dkr|d dS|r|j j||j j }tj d |tj d ||jdtdd|j jD}tj d|t|dddd} xt|j jD]\} \} } | j| | |krqhtj d| yt|j j| } WnXtk r}z:|t|g} |r|j j| | t|j j| } WYdd}~XnX|j| | |qhW| j|rV|dkrV|dj||d|j _|j jtjd|rX| rXt |t |j!kr|dtj"dt |tj"dt |j!n tjdd}d}xB|j!j#D]4\}}|j$||}||krtj%|t&|||qWxR|j#D]F\}}||j!kr&q|j!j$||}||krtj%|t&|||qW|rn|j'|j(|j)r|rtjdn tj"d n tjd! p|S)"zCheck repository consistency This method verifies all segment checksums and makes sure the index is consistent with the data stored in the segments. z is in append-only modeFcsdtj|dS)NT)r^r)r) error_foundr+r, report_errorsz&Repository.check..report_errorzStarting repository checkz&Read committed index of transaction %dNz#Failed to read committed index (%s)zNo segments transaction foundz1No index transaction found, trying latest segmentz'This repository contains no valid data.zSegment transaction is %szDetermined transaction is %scss|] }dVqdS)r Nr+)rr!r+r+r,rsz#Repository.check..zFound %d segmentszChecking segments %3.1f%%g?zrepository.check)r rr r zchecking segment file %s...zAdding commit tag to segment {}r zStarting repository index checkzIndex object count mismatch.zcommitted index: %d objectszrebuilt index: %d objectszIndex object count match.z5ID: %-64s rebuilt index: %-16s committed index: %-16sz z6Completed repository check, errors found and repaired.z)Completed repository check, errors found.z.Completed repository check, no problems found.)*rMrrAr^rrGrUrrrrrCrrrr[rr"r#rr$rrrrrrecover_segmentr%rrrrrrEr iteritemsrr_rrrr)r9Zrepairrr,rZ current_indexrrr&rr'rrr(rZ line_formatZ not_foundrvalueZ current_valuer+)r+r,checks                   "           zRepository.checkccsx|jjD]z\}}y6x0|jj|ddD]\}}}}|||||fVq(WWq tk r}ztjd||t|fWYdd}~Xq Xq WdS)aVery low level scan over all segment file entries. It does NOT care about what's committed and what not. It does NOT care whether an object might be deleted or superseded later. It just yields anything it finds in the segment files. This is intended as a last-resort way to get access to all repo contents of damaged repos, when there is uncommitted, but valuable data in there... T)r z6Segment %d (%s) has IntegrityError(s) [%s] - skipping.N)rCr#rrr^rr)r9rrrrr rurr+r+r,scan_low_levels zRepository.scan_low_levelcCs,|r|jj|jjd|_d|_d|_dS)z NF)rCr[rrErGrP)r9r[r+r+r,r`,s zRepository._rollbackcCs|jdddS)NF)r[)r`)r9r+r+r,r5szRepository.rollbackcCs |js|j|j|_t|jS)N)rErrr)r9r+r+r,__len__9szRepository.__len__cCs |js|j|j|_||jkS)N)rErr)r9r:r+r+r, __contains__>szRepository.__contains__cCs4|js|j|j|_ddt|jj|d|DS)zd list IDs starting from after id - in index (pseudo-random) order. cSsg|] \}}|qSr+r+)rid_r!r+r+r, Isz#Repository.list..)marker)rErrrr.)r9limitr6r+r+r,rCszRepository.listc Cs|dk r|dkrtd|js2|j}|j||_|dk}|rBdn|j|\}}g}x|jj|D]\}} |jj||ddd} xxyt| \} } } }Wntt fk rPYnX|dkrd}q| t kr|| f|jj | kr|j | t ||kr|SqWqbW|S)a list IDs starting from after id - in on-disk order, so that a client fetching data in this order does linear reads and reuses stuff from disk cache. We rely on repository.check() has run already (either now or some time before) and that: - if we are called from a borg check command, self.index is a valid, fresh, in-sync repo index. - if we are called from elsewhere, either self.index or the on-disk index is valid and in-sync. - the repository segments are valid (no CRC errors). if we encounter CRC errors in segment entry headers, rest of segment is skipped. Nr z$please use limit > 0 or limit = NonerF)r)r )rr)rrErrrCr#rnext StopIterationrrrrr)r9r7r6rZat_startZ start_segmentZ start_offsetresultrrZ obj_iteratorrr:r rr+r+r,scanKs.     zRepository.scanc Cs^|js|j|j|_y|j|\}}|jj|||Stk rX|j||jdYnXdS)N)rErrrCrhrr=rA)r9r:rr r+r+r,rvszRepository.getccsx|D]}|j|VqWdS)N)r)r9idsZ is_preloadedr4r+r+r,get_manys zRepository.get_manycCs|js|j|jy|j|\}}Wntk r:YnX|j|||dd|jj||\}}|jt ||jj j 7_|j j |d|j |d7<||f|j|<|jr|j|jkr|jt|jt|j|_|jdS)zput a repo object Note: when doing calls with wait=False this gets async and caller must deal with async results / exceptions later. F)update_shadow_indexrr N)rGrrrEr_deleterCrrOrrrrrrNr?rrP)r9r:ruwaitrr r+r+r,puts zRepository.putcCsd|js|j|jy|jj|\}}Wn$tk rL|j||jdYnX|j|||dddS)zdelete a repo object Note: when doing calls with wait=False this gets async and caller must deal with async results / exceptions later. NT)r>) rGrrrErrr=rAr?)r9r:r@rr r+r+r,deleteszRepository.deletecCs|r|jj|gj||j|d8<|jj|||dd}|j||7<|jj|\}}|j||7<|jj|ddS)Nr F)r)r)rFrrrrCrhrr)r9r:rr r>rr+r+r,r?szRepository._deletecCsdS)aCGet one async result (only applies to remote repositories). async commands (== calls with wait=False, e.g. delete and put) have no results, but may raise exceptions. These async exceptions must get collected later via async_response() calls. Repeat the call until it returns None. The previous calls might either return one (non-None) result or raise an exception. If wait=True is given and there are outstanding responses, it will wait for them to arrive. With wait=False, it will only return already received responses. Nr+)r9r@r+r+r,async_responseszRepository.async_responsecCsdS)z>Preload objects (only applies to remote repositories) Nr+)r9r<r+r+r,preloadszRepository.preload)FFNTFNTF)NT)F)T)T)N)FF)NN)NN)F)T)T)T)Cr'r(r)r*r r-r.r/r0r1r2r3r r4r=r>r?r8rVrWrZrbpropertyrc staticmethodrjrrrSrrrrrrrrrrrrXrTrrrrrrrrrr%rr0r1r`rr2r3rr;rr=rArBr?rCrDr+r+r+r,r&5s~D   ##    2   GF: ' d   +    r&c@sdeZdZGdddeZejdZejdks0t ejdZ e jdksHt ejdZ e jdks`t ejd Z e jd ksxt e j deZe j eeeZd>d d ZddZddZd?ddZddZddZddZddZddZd@d d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Z d,d-Z!dAd0d1Z"d2d3Z#dBd4d5Z$dCd6d7Z%dDd8d9Z&dEd:d;Z'dFd||_t||jd|_d|_||_||_d|_d|_d|_ dS)N)Zdisposer) rAr _close_fdfdsrr7rzr  _write_fd _fds_cleaned)r9rAr7rzZcapacityr+r+r,r8szLoggedIO.__init__cCs|j|jjd|_dS)N) close_segmentrLr)r9r+r+r,rTs zLoggedIO.closecCs&|\}}t|jddd|jdS)NrZDONTNEED)r rrT)r9ts_fdtsrir+r+r,rKszLoggedIO._close_fdNFc#sdkr|sdnd tjj|jd}|jtj|}|sPfdd|D}nfdd|D}t|t|d }x|D]x}tjtjj||}|sfd d|D}nfd d|D}t|t|d }x&|D]}t|tjj|||fVqWqvWdS)Nrr$r~r rucs$g|]}|jrt|kr|qSr+)rr)rdir)start_segment_dirr+r,r5sz-LoggedIO.segment_iterator..cs$g|]}|jrt|kr|qSr+)rr)rrR)rSr+r,r5s)rreversecs$g|]}|jrt|kr|qSr+)rr)rr)rr+r,r5scs$g|]}|jrt|kr|qSr+)rr)rr)rr+r,r5sll)r@rArgrzrorr)r9rrTZ data_pathdirsrR filenamesrr+)rrSr,r#s"     zLoggedIO.segment_iteratorcCs x|jddD] \}}|SWdS)NT)rT)r#)r9rrr+r+r,rszLoggedIO.get_latest_segmentcCs,x&|jddD]\}}|j|r|SqWdS)z+Return the last committed segment. T)rTN)r#is_committed_segment)r9rrr+r+r,r s z$LoggedIO.get_segments_transaction_idcCsh|d|_d}xF|jddD]6\}}||krP||jkr>|j|=t||d7}qPqWtjd||dS)z:Delete segment files left by aborted transactions r rT)rTzICleaned up %d uncommitted segment files (== everything after segment %d).N)rr#rLrr^r)r9rrrrr+r+r,r[s   zLoggedIO.cleanupc (Cs y|j|}Wntk r"dSXt|j|dn}y|j|jj tjWn6t k r}z|j t j krpdS|WYdd}~XnX|j |jj|j krdSWdQRXd}xZyt|\}}}} Wn(tk rdStk rPYnX|tkrd}q|rdSqW|S)z4Check if segment ends with a COMMIT_TAG tag FreNT)rrrXsegment_filenameseek header_fmtrr@SEEK_ENDr\r]ZEINVALrhCOMMITr8r9r) r9riteratorrirZ seen_commitrrr r!r+r+r,rW%s6  zLoggedIO.is_committed_segmentcCs"tjj|jdt||jt|S)Nru)r@rArgrrz)r9rr+r+r,rXEszLoggedIO.segment_filenamecCs| r*|jr*|j|jkr*|r"|j|j|js|j|jdkrtjj |jdt |j|j}tjj |stj |t tjj |jdt|j|jdd|_|jjtt|_|j|jkr|j|j=|jS)NrruT)r)r r7rrOrMrrzr@rArgrrrrrrXrMAGIC MAGIC_LENrL)r9Zno_newrdirnamer+r+r, get_write_fdHs      zLoggedIO.get_write_fdc sptjfdd}fdd}|yj\}}Wntk rX|}YnXjj|f|S)Ncs"tjd}|fj<|S)Nre)rXrXrL)ri)nowrr9r+r,open_fdcsz LoggedIO.get_fd..open_fdcsTjtdkrP_x6tjjD]$\}}|\}}|tkr(j|=q(WdS)Nr)rNZ FD_MAX_AGErrLr)krPrQri)rbr9r+r, clean_oldhs  z"LoggedIO.get_fd..clean_old)timeZ monotonicrLrZupd)r9rrcrerQrir+)rbrr9r,get_fd^s  zLoggedIO.get_fdcCs6|jd}|_|dk r2|jd7_d|_|jdS)Nr r)rMrr rT)r9rir+r+r,rO~s zLoggedIO.close_segmentc Cs>||jkr|j|=yt|j|Wntk r8YnXdS)N)rLrrXrl)r9rr+r+r,rs  zLoggedIO.delete_segmentcCs"|j|}tjj|o tjj|S)N)rXr@rArgetsize)r9rrr+r+r,rs zLoggedIO.segment_existscCstjj|j|S)N)r@rArhrX)r9rr+r+r,rszLoggedIO.segment_sizecCs|j|}|jd|jtS)Nr)rgrYrhr_)r9rrir+r+r,rs  zLoggedIO.get_segment_magicrTc cs|j|}|j||dkr>|jttkr:tdj|dt}|j|jj}x||r|j ||j|||t t t f|d\}}} } |r|| || fVn|| ||fV||7}|j|}|j||j|jj}qNWdS)a7 Return object iterator for *segment*. If read_data is False then include_data must be False as well. Integrity checks are skipped: all data obtained from the iterator must be considered informational. The iterator returns four-tuples of (tag, key, offset, data|size). rz-Invalid segment magic [segment {}, offset {}])r)N) rgrYrhr_r^rrrZr_readrrr) r9rr r r)riheaderrrrrur+r+r,rs$    zLoggedIO.iter_objectsc &Cstjd|||jkr |j|=tjj|t|jjkr^t |dd}|j t WdQRXdSt |dd}t |d}t j |jdt jd}t|}|}z|j t xt||jjkrT|jj|d|jj\} } } | tks&| tks&| |jjks&| t|ks&t|d| d@| kr4|d d}q|j |d| || d}qWWd~|jXWdQRXWdQRXWdQRXdS) Nzattempting to recover T)rrer)accessrlr )r^rrLr@rArhr_rZrrrr^rXmmaprZ ACCESS_READ memoryviewrrr MAX_TAG_IDr!r) r9rrriZdst_fdZsrc_fdZmmrudcrcrrr+r+r,r-s0   "( zLoggedIO.recover_segmentc Cs||jkr|jr|jj|j|}|j||j|jj}|j||j|||t f|\}}} } || krvt dj |||r~| S|S)z Read entry from *segment* at *offset* with *id*. If read_data is False the size of the entry is returned instead and integrity checks are skipped. The return value should thus be considered informational. zJInvalid segment entry header, is not for wanted id [segment {}, offset {}]) rrMsyncrgrYrhrrrirrr) r9rr r:r)rirjrrrrur+r+r,rhs   " z LoggedIO.readcCs&t|tkstdy|j|}Wn8tjk rZ} ztdj||| dWYdd} ~ XnX||jkrt|\} } } } n"||j kr|\} } } d} nt d| t krtdj| ||| |j krtdj| ||| |j }|rt|j |}t||kr tdj|||t|t|tt|ddd@| kr@td j||| dkr| ttfkr|dd |d d} }n| dkr| ttfkr|j d } |d 8}t| d krtd j||d t| |j}|j|tj|}d}||krtd j||||| |krtd j||| | | |fS)Nz7Exceeding MAX_TAG_ID will break backwards compatibilityz8Invalid segment entry header [segment {}, offset {}]: {}z$_read called with unsupported formatz?Invalid segment entry size {} - too big [segment {}, offset {}]zAInvalid segment entry size {} - too small [segment {}, offset {}]zPSegment entry data short read [segment {}, offset {}]: expected {}, got {} bytesrlz7Segment entry checksum mismatch [segment {}, offset {}]r~zOSegment entry key short read [segment {}, offset {}]: expected {}, got {} byteszPSegment entry data short seek [segment {}, offset {}]: expected {}, got {} byteszPInvalid segment entry header, did not get acceptable tag [segment {}, offset {}])maxrnrUrstructrrrrrZ TypeErrorrrrhrr!rmrrtellrYr@SEEK_CUR)r9riZfmtrjrr Zacceptable_tagsr)Z hdr_tuplerrprrrZlengthruZoldposZseekedr+r+r,risZ         $     zLoggedIO._readc Cst|}|tkr tdj|t|j|d}||jj}|j}|jj |t }|j j t |t |t |d@} |j dj| |||f|j|7_|j|fS)Nz$More than allowed put data [{} > {}])rl)rZ MAX_DATA_SIZErrrarrr header_no_crc_fmtrrcrc_fmtr!rrgr) r9r:rurZ data_sizerirr rjrpr+r+r,r)s   zLoggedIO.write_putcCsn|j|d}|jj|jjt}|jjt|t|d@}|jdj |||f|j |jj7_ |j |jjfS)N)rlrw) rarxrrrrryr!rrgr r)r9r:rrirjrpr+r+r,r7s  zLoggedIO.write_deletecCsr|r|j}|jn|j|j}|jj|jjt}|jjt |d@}|j dj ||f|j|j dS)Nlrwr ) rarqrOrxrrZrrryr!rrgr)r9rrirjrpr+r+r,r?s zLoggedIO.write_commit)rJ)NF)FF)rFT)T)T)F)F)F))r'r(r)rrrsStructrZrrUrrxryrrZ_commitr!r\r8rTrKr#rrr[rWrXrargrOrrrrrr-rhrirrrr+r+r+r,rsD           %  7  rrH)Er]rlr@rrkrsrfZbinasciirr collectionsrZ configparserrr functoolsr itertoolsrZ constantsZ hashindexr Zhelpersr r rrrrrrrrrrZlockingrrrr^rZlrucacherplatformrrrr Zalgorithms.checksumsr!Zcrypto.file_integrityr"r#r'r^rr_rrUrrrrnrrr&rrrr+r+r+r,sh