U EZh |@sddlZddlmZddlmZmZddlmZddlm Z ddl m Z m Z m Z mZddlmZmZmZmZGdd d ZGd d d ZGd d d ZGdddeZGdddeZGdddeZGddde ZGdddeZGdddeZdS)N)BytesIOreadableseekable)FULL_OBJECT_CHECKSUM_ARGS)IN_MEMORY_UPLOAD_TAG)CompleteMultipartUploadTaskCreateMultipartUploadTaskSubmissionTaskTask)ChunksizeAdjusterDeferredOpenFile get_callbacksget_filtered_dictc@s.eZdZd ddZddZddZdd Zd S) AggregatedProgressCallbackcCs||_||_d|_dS)aAggregates progress updates for every provided progress callback :type callbacks: A list of functions that accepts bytes_transferred as a single argument :param callbacks: The callbacks to invoke when threshold is reached :type threshold: int :param threshold: The progress threshold in which to take the aggregated progress and invoke the progress callback with that aggregated progress total rN) _callbacks _threshold _bytes_seen)self callbacks thresholdrX/home/aprabhat/apps/x.techxrdev.in/venv/lib/python3.8/site-packages/s3transfer/upload.py__init__"s z#AggregatedProgressCallback.__init__cCs&|j|7_|j|jkr"|dSN)rr_trigger_callbacks)rbytes_transferredrrr__call__2s z#AggregatedProgressCallback.__call__cCs|jdkr|dS)z@Flushes out any progress that has not been sent to its callbacksrN)rrrrrrflush7s z AggregatedProgressCallback.flushcCs"|jD]}||jdqd|_dS)N)rr)rr)rcallbackrrrr<s z-AggregatedProgressCallback._trigger_callbacksN)r)__name__ __module__ __qualname__rrr rrrrrr!s rc@sLeZdZdZddZdddZddd Zd d Zd d ZddZ ddZ dS)InterruptReaderaWrapper that can interrupt reading using an error It uses a transfer coordinator to propagate an error if it notices that a read is being made while the file is being read from. :type fileobj: file-like obj :param fileobj: The file-like object to read from :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The transfer coordinator to use if the reader needs to be interrupted. cCs||_||_dSr)_fileobj_transfer_coordinator)rfileobjtransfer_coordinatorrrrrPszInterruptReader.__init__NcCs|jjr|jj|j|Sr)r' exceptionr&read)ramountrrrr+TszInterruptReader.readrcCs|j||dSr)r&seek)rwherewhencerrrr-^szInterruptReader.seekcCs |jSr)r&tellrrrrr0aszInterruptReader.tellcCs|jdSr)r&closerrrrr1dszInterruptReader.closecCs|Srrrrrr __enter__gszInterruptReader.__enter__cOs |dSr)r1)rargskwargsrrr__exit__jszInterruptReader.__exit__)N)r) r"r#r$__doc__rr+r-r0r1r2r5rrrrr%Bs  r%c@sfeZdZdZdddZeddZddZd d Zd d Z d dZ ddZ ddZ ddZ ddZdS)UploadInputManageraJBase manager class for handling various types of files for uploads This class is typically used for the UploadSubmissionTask class to help determine the following: * How to determine the size of the file * How to determine if a multipart upload is required * How to retrieve the body for a PutObject * How to retrieve the bodies for a set of UploadParts The answers/implementations differ for the various types of file inputs that may be accepted. All implementations must subclass and override public methods from this class. NcCs||_||_||_dSr)_osutilr'_bandwidth_limiterrosutilr)bandwidth_limiterrrrr~szUploadInputManager.__init__cCs tddS)aDetermines if the source for the upload is compatible with manager :param upload_source: The source for which the upload will pull data from. :returns: True if the manager can handle the type of source specified otherwise returns False. zmust implement _is_compatible()NNotImplementedErrorclsZ upload_sourcerrr is_compatibles z UploadInputManager.is_compatiblecCs tddS)aWhether the body it provides are stored in-memory :type operation_name: str :param operation_name: The name of the client operation that the body is being used for. Valid operation_names are ``put_object`` and ``upload_part``. :rtype: boolean :returns: True if the body returned by the manager will be stored in memory. False if the manager will not directly store the body in memory. z%must implement store_body_in_memory()Nr=roperation_namerrrstores_body_in_memorys z(UploadInputManager.stores_body_in_memorycCs tddS)zProvides the transfer size of an upload :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request z&must implement provide_transfer_size()Nr=rtransfer_futurerrrprovide_transfer_sizesz(UploadInputManager.provide_transfer_sizecCs tddS)aDetermines where a multipart upload is required :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: boolean :returns: True, if the upload should be multipart based on configuration and size. False, otherwise. z*must implement requires_multipart_upload()Nr=rrFconfigrrrrequires_multipart_uploads z,UploadInputManager.requires_multipart_uploadcCs tddS)aReturns the body to use for PutObject :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: s3transfer.utils.ReadFileChunk :returns: A ReadFileChunk including all progress callbacks associated with the transfer future. z$must implement get_put_object_body()Nr=rErrrget_put_object_bodys z&UploadInputManager.get_put_object_bodycCs tddS)aYields the part number and body to use for each UploadPart :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type chunksize: int :param chunksize: The chunksize to use for this upload. :rtype: int, s3transfer.utils.ReadFileChunk :returns: Yields the part number and the ReadFileChunk including all progress callbacks associated with the transfer future for that specific yielded part. z)must implement yield_upload_part_bodies()Nr=)rrF chunksizerrryield_upload_part_bodiessz+UploadInputManager.yield_upload_part_bodiescCs*t||j}|jr&|jj||jdd}|S)NF)Zenabled)r%r'r9Zget_bandwith_limited_stream)rr(rrr _wrap_fileobjs z UploadInputManager._wrap_fileobjcCst|d}|rt|gSgS)Nprogress)rr)rrFrrrr_get_progress_callbackss  z*UploadInputManager._get_progress_callbackscCsdd|DS)NcSsg|] }|jqSr)r ).0r!rrr sz;UploadInputManager._get_close_callbacks..r)rZaggregated_progress_callbacksrrr_get_close_callbackssz'UploadInputManager._get_close_callbacks)N)r"r#r$r6r classmethodrArDrGrJrKrMrNrPrSrrrrr7ns   r7c@sdeZdZdZeddZddZddZdd Zd d Z d d Z ddZ ddZ ddZ ddZdS)UploadFilenameInputManagerzUpload utility for filenamescCs t|tSr) isinstancestrr?rrrrAsz(UploadFilenameInputManager.is_compatiblecCsdS)NFrrBrrrrDsz0UploadFilenameInputManager.stores_body_in_memorycCs|j|j|jjjdSr)metarGr8Z get_file_size call_argsr(rErrrrGsz0UploadFilenameInputManager.provide_transfer_sizecCs|jj|jkSr)rXsizemultipart_thresholdrHrrrrJsz4UploadFilenameInputManager.requires_multipart_uploadcCsJ||\}}||}||}||}|jj}|jj|||||dS)Nr( chunk_sizefull_file_sizerclose_callbacks)&_get_put_object_fileobj_with_full_sizerNrPrSrXrZr8#open_file_chunk_reader_from_fileobj)rrFr( full_sizerr_rZrrrrKs   z.UploadFilenameInputManager.get_put_object_bodyc cs|jj}|||}td|dD]j}||}||}||d}|j|jjj|||d\} } | | } |j j | || ||d} || fVq"dS)N) start_byte part_sizer^r\) rXrZ_get_num_partsrangerPrS'_get_upload_part_fileobj_with_full_sizerYr(rNr8ra) rrFrLr^Z num_parts part_numberrr_rdr(rbZread_file_chunkrrrrMs*     z3UploadFilenameInputManager.yield_upload_part_bodiescCst|||jjd}|S)N)Z open_function)r r8open)rr(rdrrr_get_deferred_open_file2s z2UploadFilenameInputManager._get_deferred_open_filecCs"|jjj}|jj}||d|fS)Nr)rXrYr(rZrkrrFr(rZrrrr`8s zAUploadFilenameInputManager._get_put_object_fileobj_with_full_sizecKs |d}|d}||||fS)Nrdr^)rk)rr(r4rdrbrrrrh=szBUploadFilenameInputManager._get_upload_part_fileobj_with_full_sizecCstt|jjt|Sr)intmathceilrXrZfloat)rrFrerrrrfBsz)UploadFilenameInputManager._get_num_partsN)r"r#r$r6rTrArDrGrJrKrMrkr`rhrfrrrrrUs rUc@s<eZdZdZeddZddZddZdd Zd d Z d S) UploadSeekableInputManagerz&Upload utility for an open file objectcCst|ot|Srrr?rrrrAIsz(UploadSeekableInputManager.is_compatiblecCs|dkr dSdSdS)N put_objectFTrrBrrrrDMsz0UploadSeekableInputManager.stores_body_in_memorycCsD|jjj}|}|dd|}|||j||dS)Nr)rXrYr(r0r-rG)rrFr(Zstart_positionZ end_positionrrrrGSs   z0UploadSeekableInputManager.provide_transfer_sizecKs||d}t|t|fS)Nre)r+rlen)rr(r4datarrrrh`szBUploadSeekableInputManager._get_upload_part_fileobj_with_full_sizecCs"|jjj}||jj}||fSr)rXrYr(r0rZrlrrrr`os zAUploadSeekableInputManager._get_put_object_fileobj_with_full_sizeN) r"r#r$r6rTrArDrGrhr`rrrrrqFs  rqcsheZdZdZdfdd ZeddZddZd d Zd d Z d dZ ddZ dddZ ddZ ZS)UploadNonSeekableInputManagerz7Upload utility for a file-like object that cannot seek.Ncst|||d|_dS)N)superr _initial_datar: __class__rrrzsz&UploadNonSeekableInputManager.__init__cCst|Sr)rr?rrrrA~sz+UploadNonSeekableInputManager.is_compatiblecCsdS)NTrrBrrrrDsz3UploadNonSeekableInputManager.stores_body_in_memorycCsdSrrrErrrrGsz3UploadNonSeekableInputManager.provide_transfer_sizecCsT|jjdk r|jj|jkS|jjj}|j}|||d|_t|j|krLdSdSdS)NFT)rXrZr[rYr(_readryrt)rrFrIr(rrrrrJs  z7UploadNonSeekableInputManager.requires_multipart_uploadcCs@||}||}|jjj}||j|||}d|_|Sr)rPrSrXrYr( _wrap_dataryr+)rrFrr_r(bodyrrrrKs    z1UploadNonSeekableInputManager.get_put_object_bodyc cs^|jjj}d}||}||}|d7}|||}|s s$   !,{]1