mirror of
https://github.com/aria2/aria2.git
synced 2026-04-13 16:23:03 +00:00
2fb9b5be97
To rewrite segment download mechanism for HTTP/FTP download. Use BitfieldMan to manage segment download. * src/HttpResponseCommand.h (executeInternal): Pass the reference of segment. * src/AbstractCommand.cc (prepareForRetry): Call segmentMan->cancelSegment here. (onAbort): Call segmentMan->cancelSegment here. * src/HttpDownloadCommand.cc (prepareForNextSegment): New function. * src/DownloadEngineFactory.cc (newConsoleEngine): Removed splitter. (newTorrentConsoleEngine): Removed splitter. * src/Request.h (segment): Renamed from seg. * src/FtpInitiateConnectionCommand.h (executeInternal): Pass the reference of segment. * src/AbstractCommand.h (executeInternal): Pass the reference of segment. * src/pref.h (PREF_SEGMENT_SIZE): New definition. * src/HttpProxyRequestCommand.h (executeInternal): Pass the reference of segment. * src/HttpResponseCommand.cc (checkResponse): Allowed status 206 when a request range starts 0. (handleDefaultEncoding): Rewritten the code related to Segment. (handleOtherEncoding): Rewritten the code related to Segment. * src/SegmentMan.h (SegmentEntry): New class. (SegmentEntries): New type definition. (bitfield): New variable. (usedSegmentEntries): New variable. (onNullBitfield): New function. (checkoutSegment): New function. (segments): Removed. (splitter): Removed. (unregisterId): Removed. (getSegment): New function(overload) (getDownloadedSize): Removed. (cancelSegment): New function. (completeSegment): New function. (initBitfield): New function. (hasSegment): New function. (getDownloadLength): New function. * src/BitfieldMan.h (getStartIndex): New function. (getEndIndex): New function. (getMissingUnusedIndex): New function(overload). (getSparseMissingUnusedIndex): New function. * src/BitfieldMan.cc (getMissingIndexRandomly): Handle the last byte of bitfield properly. (getMissingUnusedIndex): New function(overload). (Range): New class. (getStartIndex): New function. (getEndIndex): New function. (getSparseMissingUnusedIndex): New function. (isBitSetInternal): Return false if the given index is less than 0. * src/HttpInitiateConnectionCommand.h (executeInternal): Pass the reference of segment. * src/FtpNegotiateCommand.h (executeInternal): Pass the reference of segment. * src/FtpNegotiateCommand.cc (recvSize): Initialize bitfield here. * src/FtpTunnelResponseCommand.h (executeInternal): Pass the reference of segment. * src/HttpConnection.cc (createRequest): Rewritten range header processing. * src/DownloadCommand.h (executeInternal): Pass the reference of segment. (prepareForRetry): Removed. (prepareForNextSegment): Added an argument segment. Made it a virtual function. * src/main.cc (main): Set the initial value of PREF_SEGMENT_SIZE to 1MB. * src/SegmentMan.cc (SegmentMan): Added bitfield. Removed splitter. (~SegmentMan): Added bitfield. Removed splitter. (unregisterId): Removed. (getSegment): Rewritten. (updateSegment): Rewritten. (save): Rewritten. (read): Rewritten. (finished): Rewritten. (getDownloadedSize): Removed. (initBitfield): New function. (FindSegmentEntryByIndex): New function object. (FindSegmentEntryByCuid): New function object. (checkoutSegment): New function. (onNullBitfield): New function. (getSegment): New function(overload). (CancelSegment): New function object. (cancelSegment): New function. (completeSegment): New function. (hasSegment): New function. (getDownloadLength): New function. * src/FtpInitiateConnectionCommand.cc (executeInternal): Load .aria2 file after hostname resolution finishes. * src/Segment.h: Rewritten. * src/Segment.cc (operator<<): New function. * src/HttpDownloadCommand.h (prepareForNextSegment): New function. * src/Request.cc (resetUrl): Made segment null. * src/DownloadEngine.cc (~DownloadEngine): Call cleanQueue before deleting segmentMan. * src/HttpProxyRequestCommand.h (executeInternal): Pass the reference of segment. * src/DownloadCommand.cc (executeInternal): Rewritten the code related to Segment. (prepareForRetry): Removed. (prepareForNextSegment): Rewritten. * src/FtpTunnelResponseCommand.h (executeInternal): Pass the reference of segment. To add HTTP 1.1 persistent connection support(experimental) * src/HttpRequestCommand.cc (executeInternal): Disable keep alive if it is disabled by configuration. * src/Request.h (keepAlive): New variable. (isKeepAlive): New function. (setKeepAlive): New function. * src/pref.h (PREF_HTTP_KEEP_ALIVE): New definition. * src/HttpResponseCommand.cc (executeInternal): Check the remote server supports keep alive. * src/HttpConnection.cc (createRequest): Send "Connection: close" only if keep alive is disabled. * src/main.cc (main): Set the initial value(false) of PREF_KEEP_ALIVE to false. To add max download speed limit: * src/pref.h (PREF_MAX_SPEED_LIMIT): New definition. * src/PeerInteractionCommand.cc (executeInternal): Added max download speed limit. Not tested yet. * src/SegmentMan.h (PeerStats): New type definition. (peerStats): New variable. (registerPeerStat): New function. (FindPeerStat): New function object. (getPeerStat): New function. (calculateDownloadSpeed): New function. * src/SpeedCalc.h: New class. * src/SpeedCalc.cc: New class. * src/main.cc (main): Set the initial value of PREF_MAX_SPEED_LIMIT to 0(which means the download speed is not restricted). * src/PeerStat.h: New class. * src/SegmentMan.cc (registerPeerStat): New function. (calculateDownloadSpeed): New function. * src/DownloadCommand.cc (STARTUP_IDLE_TIME): New definition. (DownloadCommand): Register peerStat to segmentMan. Call peerStat-> downloadStart. (~DownloadCommand): Call peerStat->downloadStop. (executeInternal): Added download speed limitter. Rewritten lowest speed limitter. * src/HttpConnection.cc (receiveResponse): Fixed: eohIndex[headerBuf] -> headerBuf[eohIndex]. * src/AbstractCommand.cc (resolveHostname): Throw DlAbortEx if a name resolution failes. Added hostname to the error message. * src/ConsoleDownloadEngine.cc (calculateStatistics): Initialize psize with dlSize. * src/PieceMessage.cc (receivedAction): Do not call peerInteraction->abortPiece here. (onGotWrongPiece): Call peerInteraction->abortPiece here. * src/BitfieldMan.h (clearAllUseBit): New function. (setAllUseBit): New function. * src/BitfieldMan.cc (clearAllBit): Do not clear useBitfield here. (clearAllUseBit): New function. (setAllUseBit): New function. * src/Piece.cc (clearAllBlock): Call bitfield->clearAllUseBit().
240 lines
7.1 KiB
C++
240 lines
7.1 KiB
C++
#include "BitfieldMan.h"
|
|
#include <string>
|
|
#include <cppunit/extensions/HelperMacros.h>
|
|
|
|
using namespace std;
|
|
|
|
class BitfieldManTest:public CppUnit::TestFixture {
|
|
|
|
CPPUNIT_TEST_SUITE(BitfieldManTest);
|
|
CPPUNIT_TEST(testGetBlockSize);
|
|
CPPUNIT_TEST(testGetFirstMissingUnusedIndex);
|
|
CPPUNIT_TEST(testIsAllBitSet);
|
|
CPPUNIT_TEST(testFilter);
|
|
CPPUNIT_TEST(testGetMissingIndex);
|
|
CPPUNIT_TEST(testGetSparceMissingUnusedIndex);
|
|
CPPUNIT_TEST_SUITE_END();
|
|
private:
|
|
|
|
public:
|
|
void setUp() {
|
|
}
|
|
|
|
void testGetBlockSize();
|
|
void testGetFirstMissingUnusedIndex();
|
|
void testIsAllBitSet();
|
|
void testFilter();
|
|
void testGetMissingIndex();
|
|
void testGetSparceMissingUnusedIndex();
|
|
};
|
|
|
|
|
|
CPPUNIT_TEST_SUITE_REGISTRATION( BitfieldManTest );
|
|
|
|
void BitfieldManTest::testGetBlockSize() {
|
|
BitfieldMan bt1(1024, 1024*10);
|
|
CPPUNIT_ASSERT_EQUAL(1024, bt1.getBlockLength(9));
|
|
|
|
BitfieldMan bt2(1024, 1024*10+1);
|
|
CPPUNIT_ASSERT_EQUAL(1024, bt2.getBlockLength(9));
|
|
CPPUNIT_ASSERT_EQUAL(1, bt2.getBlockLength(10));
|
|
CPPUNIT_ASSERT_EQUAL(0, bt2.getBlockLength(11));
|
|
}
|
|
|
|
void BitfieldManTest::testGetFirstMissingUnusedIndex() {
|
|
BitfieldMan bt1(1024, 1024*10);
|
|
unsigned char bitfield[2];
|
|
memset(bitfield, 0xff, sizeof(bitfield));
|
|
|
|
CPPUNIT_ASSERT_EQUAL(0, bt1.getFirstMissingUnusedIndex(bitfield, sizeof(bitfield)));
|
|
CPPUNIT_ASSERT(bt1.setUseBit(0));
|
|
CPPUNIT_ASSERT_EQUAL(1, bt1.getFirstMissingUnusedIndex(bitfield, sizeof(bitfield)));
|
|
CPPUNIT_ASSERT(bt1.unsetUseBit(0));
|
|
CPPUNIT_ASSERT_EQUAL(0, bt1.getFirstMissingUnusedIndex(bitfield, sizeof(bitfield)));
|
|
CPPUNIT_ASSERT(bt1.setBit(0));
|
|
CPPUNIT_ASSERT_EQUAL(1, bt1.getFirstMissingUnusedIndex(bitfield, sizeof(bitfield)));
|
|
|
|
for(int i = 0; i < 8; i++) {
|
|
CPPUNIT_ASSERT(bt1.setBit(i));
|
|
}
|
|
CPPUNIT_ASSERT_EQUAL(8, bt1.getFirstMissingUnusedIndex(bitfield, sizeof(bitfield)));
|
|
|
|
CPPUNIT_ASSERT_EQUAL(8, bt1.getFirstMissingUnusedIndex());
|
|
CPPUNIT_ASSERT(bt1.setUseBit(8));
|
|
CPPUNIT_ASSERT_EQUAL(9, bt1.getFirstMissingUnusedIndex());
|
|
}
|
|
|
|
void BitfieldManTest::testIsAllBitSet() {
|
|
BitfieldMan bt1(1024, 1024*10);
|
|
CPPUNIT_ASSERT(!bt1.isAllBitSet());
|
|
bt1.setBit(1);
|
|
CPPUNIT_ASSERT(!bt1.isAllBitSet());
|
|
|
|
for(int i = 0; i < 8; i++) {
|
|
CPPUNIT_ASSERT(bt1.setBit(i));
|
|
}
|
|
CPPUNIT_ASSERT(!bt1.isAllBitSet());
|
|
|
|
for(int i = 0; i < bt1.countBlock(); i++) {
|
|
CPPUNIT_ASSERT(bt1.setBit(i));
|
|
}
|
|
CPPUNIT_ASSERT(bt1.isAllBitSet());
|
|
}
|
|
|
|
void BitfieldManTest::testFilter() {
|
|
// set random seed here in order to get same random numbers.
|
|
srandom(100);
|
|
BitfieldMan btman(2, 32);
|
|
|
|
// test offset=4, length=12
|
|
btman.addFilter(4, 12);
|
|
btman.enableFilter();
|
|
unsigned char peerBt[2];
|
|
memset(peerBt, 0xff, sizeof(peerBt));
|
|
|
|
int index;
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(3, index);
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(4, index);
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(2, index);
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(6, index);
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(5, index);
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(7, index);
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(-1, index);
|
|
CPPUNIT_ASSERT_EQUAL((long long int)12, btman.getFilteredTotalLength());
|
|
|
|
// test offset=5, length=2
|
|
btman.clearAllBit();
|
|
btman.clearFilter();
|
|
btman.addFilter(5, 2);
|
|
btman.enableFilter();
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
btman.setBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(3, index);
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
btman.setBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(2, index);
|
|
index = btman.getMissingUnusedIndex(peerBt, sizeof(peerBt));
|
|
btman.setUseBit(index);
|
|
CPPUNIT_ASSERT_EQUAL(-1, index);
|
|
CPPUNIT_ASSERT_EQUAL((long long int)4, btman.getFilteredTotalLength());
|
|
CPPUNIT_ASSERT(btman.isAllBitSet());
|
|
|
|
BitfieldMan btman2(2, 31);
|
|
btman2.addFilter(0, 31);
|
|
btman2.enableFilter();
|
|
CPPUNIT_ASSERT_EQUAL((long long int)31, btman2.getFilteredTotalLength());
|
|
}
|
|
|
|
void BitfieldManTest::testGetMissingIndex() {
|
|
srandom(100);
|
|
|
|
BitfieldMan bt1(1024, 1024*256);
|
|
|
|
unsigned char bitArray[] = {
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
};
|
|
CPPUNIT_ASSERT_EQUAL(80, bt1.getMissingIndex(bitArray, 32));
|
|
|
|
unsigned char bitArray2[] = {
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0x00, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
};
|
|
|
|
CPPUNIT_ASSERT_EQUAL(80, bt1.getMissingIndex(bitArray2, 32));
|
|
|
|
unsigned char bitArray3[] = {
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0x0f,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
};
|
|
|
|
CPPUNIT_ASSERT_EQUAL(60, bt1.getMissingIndex(bitArray3, 32));
|
|
|
|
unsigned char bitArray4[] = {
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0xff,
|
|
0xff, 0xff, 0xff, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
};
|
|
|
|
CPPUNIT_ASSERT_EQUAL(0, bt1.getMissingIndex(bitArray4, 32));
|
|
|
|
unsigned char bitArray5[] = {
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
0x00, 0x00, 0x00, 0x00,
|
|
};
|
|
|
|
CPPUNIT_ASSERT_EQUAL(-1, bt1.getMissingIndex(bitArray5, 32));
|
|
|
|
}
|
|
|
|
void BitfieldManTest::testGetSparceMissingUnusedIndex() {
|
|
BitfieldMan bitfield(1024*1024, 10*1024*1024);
|
|
|
|
CPPUNIT_ASSERT_EQUAL(0, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setBit(0);
|
|
CPPUNIT_ASSERT_EQUAL(5, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setUseBit(5);
|
|
CPPUNIT_ASSERT_EQUAL(3, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setBit(3);
|
|
CPPUNIT_ASSERT_EQUAL(8, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setBit(8);
|
|
CPPUNIT_ASSERT_EQUAL(2, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setBit(2);
|
|
CPPUNIT_ASSERT_EQUAL(7, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setBit(7);
|
|
CPPUNIT_ASSERT_EQUAL(1, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setBit(1);
|
|
CPPUNIT_ASSERT_EQUAL(4, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setBit(4);
|
|
CPPUNIT_ASSERT_EQUAL(6, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setBit(6);
|
|
CPPUNIT_ASSERT_EQUAL(9, bitfield.getSparseMissingUnusedIndex());
|
|
bitfield.setBit(9);
|
|
CPPUNIT_ASSERT_EQUAL(-1, bitfield.getSparseMissingUnusedIndex());
|
|
}
|